RAM- and CPU-Efficient Data Processing

Libraries, file formats and data types

Tommy Carstensen

Why optimize?

The problem

  • 🖥️ Shared servers, fixed RAM
  • 📈 Data exceeds memory
  • 🐘 Tools load it all at once

Three levers, same hardware

  • 🐻‍❄️🦆 The right library – stream instead of loading everything (polars, duckdb)
  • 📦 The right file format – parquet reads only the columns you ask for
  • 🔢 The right data types – store each column at the width it needs

Libraries compared

Library Language Streaming Parallel Windows / Pivots SQL
pandas 🐼 Python ❌ ❌ ✅
dask Python ✅ distributed ✅
polars 🐻‍❄️ Python ✅ in-process ✅ (✅)
duckdb 🦆 Python ✅ in-process ✅ ✅
pyarrow 🏹 Python ✅ in-process ❌
data.frame R ❌ ❌ ✅
duckdb 🦆 R ✅ in-process ✅ ✅
arrow 🏹 R ✅ in-process ❌ 
# window: a running total within each patient (row order matters)
df["cum_cost"] = df.groupby("id")["cost"].cumsum()

# pivot: reshape long -> wide, one row per year, one column per region
df.pivot_table(index="year", columns="region", values="cost", aggfunc="sum")

Same file, two ways to read it

Why streaming matters

Switching is easy

import pandas as pd
df = pd.read_parquet("events.parquet")
out = df.groupby("region")["cost"].sum()

polars

import polars as pl
df = pl.read_parquet("events.parquet")
out = df.group_by("region").agg(pl.sum("cost"))

import duckdb
out = duckdb.sql("SELECT region, sum(cost) FROM 'events.parquet' GROUP BY region")

dask

import dask.dataframe as dd
df = dd.read_parquet("events.parquet")
out = df.groupby("region")["cost"].sum().compute()

Read less, store smart

File formats

Format Col select Peek Pushdown Nested Metadata Typing Python R
CSV ❌ ❌ ❌ ❌ ❌ ❌ ✅ ✅
SAS ✅ ✅ ❌ ❌ ✅ ✅ ✅ ✅
HDF5 ✅ ✅ ❌ ✅ ✅ (✅) ✅ ✅
SQLite ✅ ✅ ✅ ❌ (✅) (✅) ✅ ✅
.rds ❌ ❌ ❌ ✅ ❌ ✅ ❌ ✅
qs2 ❌ ❌ ❌ ✅ ❌ ✅ ❌ ✅
fst ✅ ✅ ❌ ❌ ❌ (✅) ❌ ✅
feather ✅ ✅ ❌ ✅ ✅ ✅ (✅) ✅
parquet ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅

( ) = partial. Typing: primitives kept, factors + dates coerced. Metadata: limited. Python: via pyarrow.

How parquet reads less

Column select

Metadata peek

Filter pushdown

polars’ flat bar is a known gap on float columns – reported upstream as pola-rs/polars#27860 (accepted), with a nan_count-gated fix verified locally. It skips fine on integer and date columns.

Data types

Type Min size NaN? Memory Fast ops Parquet
bool 1 bit ❌ ✅✅ ✅✅ ✅
int8/16/32/64 1 / 2 / 4 / 8 B ❌ ✅ ✅ ✅
float32/64 4 / 8 B ✅ ✅ ✅ ✅
datetime64 8 B ✅ ✅ ✅ ✅
str / object ~50 B ✅ ❌ ❌ ✅
category 1 B ✅ ✅✅ ✅✅ ✅

Data types: speed

Categorical is deliberately off this chart: the engines order it differently – pandas sorts by the integer codes (fast), polars sorts lexically since 1.32 (string speed), duckdb decodes on read, and pyarrow sorts a dictionary array but not a dictionary column of a table (ArrowNotImplementedError, verified on pyarrow 25). For a key you sort on, store an integer code; keep categorical for group-by and RAM.

Data types: RAM

The recipe

On one large extract, the levers stack:

  • 🐻‍❄️🦆 Stream with polars or duckdb, not pandas – RAM stays flat as rows grow
  • 📦 Store it as parquet with the right dtypes – small in RAM to begin with
  • 📊 Read only the columns you need – less I/O
  • 👀 Peek the footer instead of scanning – schema and row count instantly

You don’t need a bigger machine

  • 💻 The recipe turns a pandas-OOM job into one that runs in ~16 GB
  • 📈 Streaming even reaches model fitting – Cox & GLM at flat RAM (appendix)

800 million rows grouped in under 0.5 GB of peak RAM. No high-RAM server, no cloud rental.

Appendix

Streaming taken all the way to model fitting: exact Cox and GLM coefficients at flat RAM, in two small open-source packages.

coxstream: streaming Cox PH

pip install coxstream     on CRAN: install.packages("coxstream")

renew_glm: streaming GLM

pip install renew-glm     Gaussian / binomial / Poisson

How memory was measured

All data is synthetic, shaped like registry / EHR extracts – no real patient data. RAM is the peak true footprint (macOS phys_footprint, Linux RSS+swap, so compressed pages are counted), not resident RSS, which under-reports the peak by 2-3x under memory pressure. Python and R figures are not directly comparable: R carries ~150 MB of interpreter overhead.

tommycarstensen.github.io