Slides

Tommy Carstensen

Talks on where memory and time actually go in large-scale health data analysis, and on making exact estimators run when the table does not fit.

Decks

RAM- and CPU-efficient data processing

Libraries, file formats and data types, measured rather than asserted. Streaming engines against pandas, parquet against CSV, and what the wrong dtype costs in both speed and memory. Python and R, 50 to 800 million rows, with the measurement methodology included.

Peak memory when grouping 100 to 800 million rows pandas needs 3.5 GB at 100 million rows and 13.5 GB at 400 million, then runs out of memory. polars streams the same query and stays between 0.29 and 0.40 GB all the way to 800 million rows. 0 5 10 100 200 400 800 ROWS, MILLIONS PEAK RAM, GB 3.5 6.8 13.5 pandas loads the whole table, out of memory past 400 M out of memory past 400 M polars streams, 0.29 to 0.40 GB

open the deck