Apache Parquet: Advantages and Log Use Case Analysis

Background

I recently came across several articles about log solutions and noticed they all use Apache Parquet as their storage file format:

• Yelp Releases Solution for Managing S3 Server Access Logs at Scale - Architecture - InfoQ
• Cloudflare Log Explorer is now GA, providing native observability and forensics
• Cost Reduction Against the Trend: Governance Practices for 30% Annual Reduction on Cloud Data Platforms - Cloud Computing - InfoQ
• Object Storage Is All You Need - Justin Cormack, Docker
• Honestly, OpenObserve Is Pretty Good!
• AWS Debuts a Distributed SQL Database, Amazon S3 Tables for Iceberg - The New Stack
• Grafana Tempo 2.5 release: vParquet4, streaming endpoints, and more metrics | Grafana Labs
• Object Store Apps: Cloud Native’s Freshest Architecture - The New Stack

This piqued my curiosity:

• What is Apache Parquet?
• What are its advantages?
• What software can process Apache Parquet?
• Why are so many log solutions converting logs to Apache Parquet, and what benefits does this bring?

Introduction to Apache Parquet

Apache Parquet is an open-source columnar storage file format designed specifically for big data processing frameworks. Originally co-developed by Twitter and Cloudera, it is now an Apache top-level project.

Core Advantages

1. Columnar Storage Structure

• Unlike traditional row-based storage, Parquet stores data by column
• Queries only need to read relevant columns, significantly reducing I/O
• Comparison example:

Row-based storage: Row1[col1,col2,col3], Row2[col1,col2,col3], ...
Columnar storage:  Column1[all row values], Column2[all row values], ...

2. Efficient Compression and Encoding

• Data within the same column shares a consistent type, enabling much higher compression ratios (up to 1/10 of row-based storage)
• Supports multiple encodings: RLE, dictionary encoding, delta encoding, etc.
• Supports multiple compression algorithms: Snappy, Gzip, LZO, Zstd

3. Schema Evolution Support

• Supports backward/forward-compatible schema changes
• Columns can be added, removed, or have their types modified

4. Predicate Pushdown

• Query engines can filter out irrelevant data blocks before reading the data
• Leverages column statistics (min/max values) to skip unrelated data blocks

5. Nested Data Structure Support

• Natively supports complex nested data types (arrays, maps, structs)
• Uses the Dremel record shredding algorithm for efficient nested data storage

Software/Frameworks That Handle Parquet

Big Data Processing Frameworks

• Apache Spark (primary use case)
• Apache Hive
• Apache Impala
• Presto/Trino
• Apache Flink
• Apache Arrow (in-memory format conversion)

Query Engines

• AWS Athena
• Google BigQuery
• Azure Synapse
• DuckDB
• Polars

Programming Language Support

• Python (PyArrow, pandas)
• Java
• R
• Go
• .NET

Log Solutions

• Cloudflare Log Explorer
• OpenObserve
• Grafana Tempo
• Yelp
• AWS official reference architecture: Extracting key insights from Amazon S3 access logs with AWS Glue for Ray | AWS Big Data Blog

Why Log Solutions Are Adopting Parquet

1. Cost Efficiency

# Example: Log storage cost comparison
Raw JSON logs:        1TB → Storage cost $$$$
After Parquet compression: ~100GB → Storage cost $

• Storage costs reduced by 70-90%
• Network transfer costs significantly lowered

2. Query Performance Improvement

-- Typical log query scenario
SELECT COUNT(*), error_code 
FROM logs 
WHERE date >= '2024-01-01' 
  AND status = 'ERROR' 
GROUP BY error_code;

-- Parquet advantages:
-- 1. Only reads the date, status, and error_code columns
-- 2. Uses column statistics to quickly skip irrelevant date partitions
-- 3. Compressed data reduces disk I/O

3. Well-Suited for Time-Series Data Analysis

• Log data inherently has a temporal dimension
• Parquet supports time-based partitioning, optimizing time-range queries
• Combined with partition pruning, performance improves dramatically

4. Compatible with the Modern Data Stack

# Typical log processing pipeline
Raw logs → Fluentd/Logstash → Kafka → 
Spark Streaming → Parquet (S3/ADLS) → 
Trino/Athena queries → BI tools

5. Long-Term Storage and Analysis

• Parquet is the ideal format for analytical workloads
• Supports data lake architectures (Delta Lake, Iceberg, Hudi)
• Facilitates trend analysis and machine learning on historical logs

Practical Use Case Examples

Case: ELT Log Analytics Pipeline

Raw logs (JSON/text)
       ↓
Real-time processing layer (Kafka)
       ↓
Batch processing layer (Spark) → Convert to Parquet
       ↓
Cloud storage (S3/GCS) → Partitioned: dt=2024-01-01/
       ↓
Query layer (Athena/Presto)
       ↓
Visualization (Grafana/Tableau)

Performance Comparison

• Storage space: 75-90% reduction compared to JSON
• Query speed: 10-100x improvement (depending on query patterns)
• Data scanned: 60-95% reduction (column pruning effect)

Caveats

1. Scenarios where Parquet is NOT ideal:

   • High-frequency single-row reads/writes (OLTP)
   • Scenarios requiring streaming row-by-row processing
   • Too many small files can degrade performance

2. Best practices:

   • Set appropriate file sizes (128MB-1GB)
   • Organize data with time-based partitioning
   • Choose a suitable compression algorithm (balance speed vs. ratio)

Parquet has become the de facto standard format for modern data lakes and log analytics, particularly well-suited for log management scenarios requiring long-term storage, batch analysis, and cost optimization.