📐 Parquet vs CSV Compression
Your 500 GB CSV dataset becomes ~60 GB in Parquet with Snappy compression. That is not a typo — it is an 8× reduction. Columnar format + predicate pushdown means queries that scanned 500 GB now touch 8 GB. Storage savings are the appetizer. Query performance is the main course.
📐 Dataset Configuration
Input raw CSV size. Pick your codec ratio — Snappy=0.22, zstd=0.15, gzip=0.10. Output: compressed Parquet size, storage saved, and annual S3 cost for both formats. Because the finance team wants the dollar number, not the compression ratio.
Parquet vs CSV: Columnar Compression Achieves 78–90% Reduction
CSV (Comma-Separated Values) is the universal interchange format for tabular data — human-readable, universally supported, and catastrophically inefficient for large-scale analytics. A 150 GB CSV file, when converted to Apache Parquet with Snappy compression, routinely shrinks to 30-35 GB — a 78-80% reduction. With Zstandard (zstd) compression at its default level, the reduction reaches 85%. With Gzip compression, Parquet files can achieve 90% compression ratios at the cost of slower decompression during query execution. This is not a marginal improvement — it is a fundamental architectural difference between row-oriented and columnar data representation.
Why Columnar Layout Compresses So Aggressively
CSV stores data row by row: each line is a complete record with all columns, separated by delimiters. Parquet stores data column by column: all values of column A are stored together, then all values of column B, and so on. This columnar layout is the foundation for massive compression gains, because values within a single column tend to be highly homogeneous — similar data types, limited ranges, and many repeated values. Compression algorithms exploit this homogeneity ruthlessly:
| Technique | How It Works | Best For |
|---|---|---|
| Dictionary Encoding | Replaces repeated values with small integer indices. A column with 1M rows but only 50 distinct values stores a 50-entry dictionary plus 1M tiny indices. | Low-cardinality columns (status, category, country) |
| Run-Length Encoding (RLE) | Stores consecutive identical values as (value, count) pairs. "AAAAABBBBB" becomes "A×5, B×5". | Sorted or pre-grouped columns |
| Delta Encoding | Stores the difference between consecutive values rather than the values themselves. Timestamps: "0, +5, +3, +2" instead of "1700000000, 1700000005, 1700000008, 1700000010". | Monotonically increasing columns (timestamps, IDs) |
| Bit-Packing | Packs small integers into the minimum number of bits per value. A column of values 0-7 uses 3 bits per value instead of 32. | Integer columns with small ranges |
These techniques are applied sequentially in Parquet's encoding pipeline, then the resulting data blocks are compressed with a general-purpose codec (Snappy, zstd, or Gzip) for a final pass. The compounding effect is what delivers the 78-90% reduction observed in production data lakes.
Real Compression Ratios at Scale
Benchmark data from production data lakes reveals consistent ratios across workloads. Snappy (the default and most widely used codec in the Hadoop/Spark ecosystem) achieves a 0.18-0.25 ratio on structured log and event data, meaning a 100 GB CSV dataset compresses to 18-25 GB. Zstandard at compression level 3 achieves 0.12-0.18 (82-88% reduction) with decompression speeds comparable to Snappy — making it the preferred codec for query-heavy workloads. Gzip at level 6 achieves 0.08-0.12 (88-92% reduction) but decompresses 3-5× slower, making it suitable for archival/cold storage tiers. LZ4 and LZO offer the fastest decompression but weaker compression (0.25-0.35 ratio), useful for interactive ad-hoc query workloads where CPU is the bottleneck. The codec ratio input on this calculator represents the fraction of original CSV size that the Parquet output will occupy.
S3 Cost Impact: The Financial Case for Parquet
At AWS S3 Standard pricing of $0.023/GB/month, a 150 GB CSV dataset costs approximately $41.40/year to store. The equivalent Parquet dataset at a 0.22 compression ratio occupies only 33 GB, costing $9.11/year — a savings of $32.29/year. For a single dataset, this seems modest. But a typical enterprise data lake contains hundreds of terabytes to petabytes of CSV-derived data. At 1 PB of raw CSV, the annual S3 cost is approximately $275,000. Converting to Parquet at 0.22 ratio reduces the bill to $60,700 — a $214,300 annual savings on storage alone, before accounting for reduced data transfer costs, faster query execution, and lower compute spend from scanning less data. The return on investment for a one-time Parquet conversion is measured in weeks, not months.