Uploaded November 2020 | Updated September 2026, 2 hours ago
This is part 2 on automating the import of CSV files to a database - a common data science task . In this video, we'll take our python script from part 1, which manually imports a CSV/Excel file to a postgres database, and completely automate the script so that it can import multiple files without the need to edit the code. You'd essentially be saving hours of work.
Importing CSV files to a database is a common task needed for data science and analytics and it can be done completely with python using pandas dataframes, numpy, os library, and the posgres database wrapper psycopg2.
This video focuses on automation. If you're interested in build a script to import files to a database, check out the first video in our series: youtu.be/wqBFgaMgFQA.
Topics Covered:
- Automatically identifying your CSVs and handling multiple files using for loops, bash commands, and the os library
- Automatically cleaning the table name and column headers using methods like replace() and lower() using format() and %s variables
- Automatically converting your pandas dataframe to a CSV file using for loops and dictionaries
- Automatically creating multiple database tables and importing your data to a postgres database on Amazon Web Services (AWS) in python
______________________________________________________________________
Resources:
- How to setup Anaconda: docs.anaconda.com/anaconda/install
- How to install psygopg2: anaconda.org/anaconda/psycopg2
👉 Subscribe to my channel: bit.ly/2GsFxmA
👉 Practice real data science interview questions: platform.stratascratch.com/coding?utm_source=youtube&utm_medium=click&utm_campaign=YT+description+link
______________________________________________________________________
Timestamps:
Intro: (0:00)
Identifying CSVs automatically: (2:20)
Automatically making a new directory and moving files: (6:55)
Converting CSV files to pandas dataframes: (13:13)
Automating the cleaning of table names and headers: (18:17)
Automatically connect to database and import files: (25:17)
______________________________________________________________________
Contact:
If you have any questions, comments, or feedback, please leave them here!
Feel free to also email me at nathan@stratascratch.com
______________________________________________________________________
This is part 2 on automating the import of CSV files to a database - a common data science task . In this video, we'll take our python script from part 1, which manually imports a CSV/Excel file to a postgres database, and completely automate the script so that it can import multiple files without the need to edit the code. You'd essentially be saving hours of work.
Importing CSV files to a database is a common task needed for data science and analytics and it can be done completely with python using pandas dataframes, numpy, os library, and the posgres database wrapper psycopg2.
This video focuses on automation. If you're interested in build a script to import files to a database, check out the first video in our series: youtu.be/wqBFgaMgFQA.
Topics Covered:
- Automatically identifying your CSVs and handling multiple files using for loops, bash commands, and the os library
- Automatically cleaning the table name and column headers using methods like replace() and lower() using format() and %s variables
- Automatically converting your pandas dataframe to a CSV file using for loops and dictionaries
- Automatically creating multiple database tables and importing your data to a postgres database on Amazon Web Services (AWS) in python
______________________________________________________________________
Resources:
- How to setup Anaconda: docs.anaconda.com/anaconda/install
- How to install psygopg2: anaconda.org/anaconda/psycopg2
👉 Subscribe to my channel: bit.ly/2GsFxmA
👉 Practice real data science interview questions: platform.stratascratch.com/coding?utm_source=youtube&utm_medium=click&utm_campaign=YT+description+link
______________________________________________________________________
Timestamps:
Intro: (0:00)
Identifying CSVs automatically: (2:20)
Automatically making a new directory and moving files: (6:55)
Converting CSV files to pandas dataframes: (13:13)
Automating the cleaning of table names and headers: (18:17)
Automatically connect to database and import files: (25:17)
______________________________________________________________________
Contact:
If you have any questions, comments, or feedback, please leave them here!
Feel free to also email me at nathan@stratascratch.com
______________________________________________________________________
