NEW YORK — What happened
A comprehensive academic and industry study audited the performance of transactional storage engines that bypass traditional proprietary database storage files entirely, writing directly to open columnar formats like Apache Parquet and Lance with full ACID (Atomicity, Consistency, Isolation, Durability) compliance.
Why it matters
For decades, companies have maintained two separate database estates: row-oriented operational databases for live user transactions, and columnar data warehouses for analytics. Moving data between the two requires complex, expensive ETL (extract, transform, load) data pipelines that introduce lag and failure points. A reliable engine that handles live transactional writes while keeping data formatted for instant analytical queries could eliminate billions of dollars in enterprise data plumbing.
Deep dive
The audit evaluated engines using NVMe-over-Fabrics write-ahead logging (WAL) paired with micro-batch compaction workers. Incoming user transactions are immediately acknowledged via low-latency persistent append-only logs. Simultaneously, local background routines convert those log records into open columnar files stored directly on distributed object storage tiers. Under moderate read and write volumes (up to 12,000 transactions per second), the audited architectures maintained sub-15-millisecond commit latencies while presenting an analytics-ready Parquet catalog to external query tools without any manual data movement.
Report check
Early architectural claims insisted that object-storage-backed systems could replace standard PostgreSQL or MySQL setups without any trade-offs. The independent audit revealed that under high-concurrency point updates—such as hundreds of concurrent users modifying identical account balances—write amplification surged by 400%, forcing compaction workers to burn excessive compute cycles and spiking storage bills.
Open questions
Can modern metadata catalogs scale safely without catastrophic lock contention when thousands of microservices perform real-time updates directly on shared object store namespaces?