Memory Error while concatenating multiple Pandas Dataframes

Question

We are trying to load IDS-2018 dataset, it consists of 10 CSV files with a total of 6.4 GB. When we tried concat all the CSV files in a 32GB RAM server, it's crashing (Process is Killed).

We even tried optimizing the storage space in a pandas data frame by using,

def reduce_mem_usage(df):
    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']
    start_mem = df.memory_usage().sum() / 1024**2
    for col in df.columns:
        col_type = df[col].dtypes
        if col_type in numerics:
            c_min = df[col].min()
            c_max = df[col].max()
            if str(col_type)[:3] == 'int':
                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:
                    df[col] = df[col].astype(np.int8)
                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
                    df[col] = df[col].astype(np.int16)
                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:
                    df[col] = df[col].astype(np.int32)
                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:
                    df[col] = df[col].astype(np.int64)
            else:
                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:
                    df[col] = df[col].astype(np.float16)
                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:
                    df[col] = df[col].astype(np.float32)
                else:
                    df[col] = df[col].astype(np.float64)
    end_mem = df.memory_usage().sum() / 1024**2
    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))
    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))
    return df

But there is no use. Server is still crashing while concatenating each of the CSV files. We have concatenated each of the files using pd.concat. whole code is here. How to achieve this so that we could do further processing??

Answer 1

I would try the following:

• Specifying column types on read_csv via the dtypes argument.
• Not creating 10 dataframes and rely on del.

import numpy as np
import pandas as pd

data_files = [
    './data/CSVs/02-14-2018.csv',
    './data/CSVs/02-15-2018.csv',
    ... # a few more
]

# define dtypes
data_types = {
  "col_a": np.float64,
  ... # other types
}

df = reduce_memory_usage(
    pd.read_csv(filename[0], dtype=data_types, index_col=False)
)
for filename[1:] in data_files:
    df = pd.concat(
        [
            df,
            reduce_mem_usage(
                pd.read_csv(
                    filename,
                    dtype=data_types,
                    index_col=False,
                )
            ),
        ],
        ignore_index=True,
    )

This way you make sure the type inference is exactly what you needed to be and reduce the memory footprint. Also, if you have categorical columns on your data which are usually encoded as strings on CSV files you can greatly reduce memory footprint by using a categorical column data type.