📐 Parquet vs CSV Compression
Your 500 GB CSV dataset becomes ~60 GB in Parquet with Snappy compression. That is not a typo — it is an 8× reduction. Columnar format + predicate pushdown means queries that scanned 500 GB now touch 8 GB. Storage savings are the appetizer. Query performance is the main course.
📐 Dataset Configuration
Input raw CSV size. Pick your codec ratio — Snappy=0.22, zstd=0.15, gzip=0.10. Output: compressed Parquet size, storage saved, and annual S3 cost for both formats. Because the finance team wants the dollar number, not the compression ratio.
Parquet vs CSV: Columnar Compression Achieves 78–90% Reduction
CSV (Comma-Separated Values) is the universal interchange format for tabular data — human-readable, universally supported, and catastrophically inefficient for large-scale analytics. A 150 GB CSV file, when converted to Apache Parquet with Snappy compression, routinely shrinks to 30-35 GB — a 78-80% reduction. With Zstandard (zstd) compression at its default level, the reduction reaches 85%. With Gzip compression, Parquet files can achieve 90% compression ratios at the cost of slower decompression during query execution. This is not a marginal improvement — it is a fundamental architectural difference between row-oriented and columnar data representation.
Why Columnar Layout Compresses So Aggressively
CSV stores data row by row: each line is a complete record with all columns, separated by delimiters. Parquet stores data column by column: all values of column A are stored together, then all values of column B, and so on. This columnar layout is the foundation for massive compression gains, because values within a single column tend to be highly homogeneous — similar data types, limited ranges, and many repeated values. Compression algorithms exploit this homogeneity ruthlessly:
| Technique | How It Works | Best For |
|---|---|---|
| Dictionary Encoding | Replaces repeated values with small integer indices. A column with 1M rows but only 50 distinct values stores a 50-entry dictionary plus 1M tiny indices. | Low-cardinality columns (status, category, country) |
| Run-Length Encoding (RLE) | Stores consecutive identical values as (value, count) pairs. "AAAAABBBBB" becomes "A×5, B×5". | Sorted or pre-grouped columns |
| Delta Encoding | Stores the difference between consecutive values rather than the values themselves. Timestamps: "0, +5, +3, +2" instead of "1700000000, 1700000005, 1700000008, 1700000010". | Monotonically increasing columns (timestamps, IDs) |
| Bit-Packing | Packs small integers into the minimum number of bits per value. A column of values 0-7 uses 3 bits per value instead of 32. | Integer columns with small ranges |
These techniques are applied sequentially in Parquet's encoding pipeline, then the resulting data blocks are compressed with a general-purpose codec (Snappy, zstd, or Gzip) for a final pass. The compounding effect is what delivers the 78-90% reduction observed in production data lakes.
Real Compression Ratios at Scale
Benchmark data from production data lakes reveals consistent ratios across workloads. Snappy (the default and most widely used codec in the Hadoop/Spark ecosystem) achieves a 0.18-0.25 ratio on structured log and event data, meaning a 100 GB CSV dataset compresses to 18-25 GB. Zstandard at compression level 3 achieves 0.12-0.18 (82-88% reduction) with decompression speeds comparable to Snappy — making it the preferred codec for query-heavy workloads. Gzip at level 6 achieves 0.08-0.12 (88-92% reduction) but decompresses 3-5× slower, making it suitable for archival/cold storage tiers. LZ4 and LZO offer the fastest decompression but weaker compression (0.25-0.35 ratio), useful for interactive ad-hoc query workloads where CPU is the bottleneck. The codec ratio input on this calculator represents the fraction of original CSV size that the Parquet output will occupy.
S3 Cost Impact: The Financial Case for Parquet
At AWS S3 Standard pricing of $0.023/GB/month, a 150 GB CSV dataset costs approximately $41.40/year to store. The equivalent Parquet dataset at a 0.22 compression ratio occupies only 33 GB, costing $9.11/year — a savings of $32.29/year. For a single dataset, this seems modest. But a typical enterprise data lake contains hundreds of terabytes to petabytes of CSV-derived data. At 1 PB of raw CSV, the annual S3 cost is approximately $275,000. Converting to Parquet at 0.22 ratio reduces the bill to $60,700 — a $214,300 annual savings on storage alone, before accounting for reduced data transfer costs, faster query execution, and lower compute spend from scanning less data. The return on investment for a one-time Parquet conversion is measured in weeks, not months.
Pricing Basis, Sources & Assumptions
Every rate on this page is a published vendor list price — no negotiated discounts, private pricing, or credit offsets. Totals are in USD and exclude tax. Rates were last checked against the sources below on .
| Pricing input | Basis used on this page |
|---|---|
| Region | US East (N. Virginia) — S3 Standard at $0.023 per GB-month, the single rate this page prices against. |
| Currency | USD — on-demand list price, tax excluded |
| Last checked | — The S3 Standard storage rate ($0.023/GB-month) and the archive-tier behaviour described in the discussion were re-checked on this date. |
Modelling assumptions
- Annual cost = stored size × $0.023/GB-month × 12. The model prices storage only.
- The compression ratio is applied to raw CSV size to get Parquet size; the codec ratios offered are typical observed values, not guarantees — a specific dataset can beat or miss them depending on column cardinality and data types.
- Encrypted, already-compressed, or high-entropy columns compress poorly, so real-world ratios for mixed workloads are often worse than the preset suggests.
- Volumes are decimal: 1 GB = 1,000 MB, and 1 PB = 1,000 TB.
- The comparison holds both datasets in the same storage class; tiering the Parquet copy to a colder class would widen the gap further.
What this model excludes
- Request costs: S3 charges per PUT, GET, and LIST, and columnar formats typically issue fewer, larger GETs. The direction of that saving is favourable but it is not modelled.
- Compute cost of the conversion job itself, and the egress incurred if it runs outside the region.
- Early-deletion charges when a lifecycle rule moves smaller objects sooner.
- Replication, versioning, and cross-region copies of the Parquet dataset.
Sources
- AWS (2026). "Amazon S3 Pricing." Per-GB-month storage rates by class, request pricing, and retrieval charges. aws.amazon.com
- AWS (2026). "Amazon S3 Storage Classes." Which classes fit which access pattern, including archive retrieval behaviour. aws.amazon.com
- AWS (2026). "S3 Intelligent-Tiering." Automatic class movement, which interacts with the storage saving modelled here. docs.aws.amazon.com
Vendor list prices change without notice — re-check the linked pages before committing spend. jslet takes no vendor sponsorship and carries no affiliate links; see about.