A Fast Path for Fixed-Length Lists in Parquet

Table of Contents Parquet’s Dremel Encoding Reading Effectively-Fixed-Length Lists Faster Performance Gains Summary In its current form Apache Parquet isn’t a great fit for storing fixed-length lists, such as coordinates, RGB(A) colors, or—​an increasingly common case—​vector embeddings driving search and retrieval workloads. A 768-dimensional embedding is just a list of floats that always has the same length, yet Parquet’s Dremel machinery encodes it as if that length could vary from row to row, spelling out and reconstructing each vector’s structure on read. That costs roughly 3× more than a purely flat columnar representation of the same data (see apache/arrow#34510).