Uploaded March 2019 | Updated September 2026, 2 weeks ago
Tired of getting Memory Errors while trying to read very big (more than 1 GB) CSV files to Python? This is a common case when you download a very rich dataset from Kaggle.com and try to load the whole dataset to you Python environment instead to put the data to the cloud as Amazon Web Server or Google Cloud. Also this is an issue of Big Data handling in Python environment.
There is a simple way how you can handle it. And the solution is to load the very big data file by iterate the reading procedure by chunksize (that depends on you computer memory amount), build a Sqlite database from this and finally extract the required data by defined SQL query (from database to Pandas dataframe).
And there are only few steps for it:
Step 1: 0:03 Set directory to origin data file (very big CSV file in my case).
Step 2: 1:31 Read the first 2 rows of origin file. This is useful to check the columns and structure of the dataset.
Step 3: 2:05 Trying to read the very big CSV file in standard way by using Pandas read method. This will cause Memory Error.
Step 4: 2:23 Solution. This part consist of 3 steps. The first one is to build a engine that is responsible to create a database (space for future data in reading procedure). The second step is to write a loop that perform the reading procedure by bunches of data which size is defined by chunksize parameter. And finally construct the Pandas dataframe by write a SQL query to the database we just built. That will create a easy readable Pandas dataframe.
Used libraries and modules:
1. pandas: pandas.pydata.org
2. numpy (not used, but still): numpy.org
3: sqlite3: docs.python.org/2/library/sqlite3.html
4. sqlalchemy: sqlalchemy.org (this library is used to build a engine for creating a database from origin very big CSV file).
Thank you for watching.
Vytautas Bielinskas
linkedin.com/in/bielinskas
#bigcsvfiles #pandas #bigdata
Tired of getting Memory Errors while trying to read very big (more than 1 GB) CSV files to Python? This is a common case when you download a very rich dataset from Kaggle.com and try to load the whole dataset to you Python environment instead to put the data to the cloud as Amazon Web Server or Google Cloud. Also this is an issue of Big Data handling in Python environment.
There is a simple way how you can handle it. And the solution is to load the very big data file by iterate the reading procedure by chunksize (that depends on you computer memory amount), build a Sqlite database from this and finally extract the required data by defined SQL query (from database to Pandas dataframe).
And there are only few steps for it:
Step 1: 0:03 Set directory to origin data file (very big CSV file in my case).
Step 2: 1:31 Read the first 2 rows of origin file. This is useful to check the columns and structure of the dataset.
Step 3: 2:05 Trying to read the very big CSV file in standard way by using Pandas read method. This will cause Memory Error.
Step 4: 2:23 Solution. This part consist of 3 steps. The first one is to build a engine that is responsible to create a database (space for future data in reading procedure). The second step is to write a loop that perform the reading procedure by bunches of data which size is defined by chunksize parameter. And finally construct the Pandas dataframe by write a SQL query to the database we just built. That will create a easy readable Pandas dataframe.
Used libraries and modules:
1. pandas: pandas.pydata.org
2. numpy (not used, but still): numpy.org
3: sqlite3: docs.python.org/2/library/sqlite3.html
4. sqlalchemy: sqlalchemy.org (this library is used to build a engine for creating a database from origin very big CSV file).
Thank you for watching.
Vytautas Bielinskas
linkedin.com/in/bielinskas
#bigcsvfiles #pandas #bigdata
