Track · 2:29 · Liner note
Parquet Floors
Parquet is an open columnar file format that stores data by column, with compression and embedded statistics, so analytical engines can read only the columns and row groups a query needs.
Parquet files store data by column, in compressed row groups that carry their own minimum and maximum statistics. An engine can skip the columns a query does not name and the row groups its filter rules out. Parquet Floors is the flooring of a lake: a dependable surface, laid once, that many tools can walk across.
Laying it well matters. Avoid thousands of tiny files, sort or partition on the columns you filter by and keep the schema consistent across files. Small files are the cracked boards of a lake.