Columnar storage layer and the .qfc file format

[sahilkalgutkar]

The foundation the rest of the engine sits on.

Arrays and batches. A column is a typed buffer plus an optional packed validity bitmap — optional because a column with no nulls should not pay for a mask, and most don't. Strings use one contiguous byte buffer plus an offsets vector rather than a String per row, so scanning a string column touches two allocations instead of a million. RecordBatch is the unit that moves between operators; nothing in the engine has a "next row" method.

The .qfc format. Magic, row groups of column chunks, then a footer holding the schema and, per chunk, its byte range, encoding and zone map. The footer is at the end with its length as the last field, so a reader seeks to len - 8 and learns the whole layout without touching data. That is what lets a query over two of forty columns read two chunks per row group, and a query whose predicate falls outside a row group's [min, max] read none of it.

Encodings are chosen by measurement, not by type. Two signals decide it — average run length and distinct/row ratio — because the right answer changes between row groups of the same column. On the tests' fixtures RLE comes out >10x smaller than plain on a sorted column, and the dictionary 4x smaller on a shuffled four-value string column; both assertions are in the suite.

Things I decided deliberately, and why:

• Equality for grouping is not equality for WHERE. Value's PartialEq/Hash are grouping semantics (two NULLs group together, Int64(7) and Float64(7.0) hash alike so an int/float hash join doesn't silently drop rows). sql_compare stays the authority for predicates and returns None for NULL.
• A zone map that guesses wrong in one direction is fine and in the other is a bug. Every branch of may_match defaults to "may match" when it cannot prove otherwise. <> only prunes a row group that is a single value with no nulls.
• Merged distinct counts are an upper bound and the code says so. Row groups can overlap and the footer can't tell; the planner only uses them to order joins, so an over-estimate costs a plan, not an answer.
• A zero-column batch still remembers its row count, or SELECT count(*) after a filter would come back zero.

Verification. 122 tests in this crate. Corrupt-input paths are tested explicitly: bad magic, truncation, a footer length past EOF, a dictionary code past its dictionary, an RLE run longer than the declared row count, a validity mask of the wrong width.

[codecov]

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

ℹ️ You can also turn on project coverage checks and project coverage reporting on Pull Request comment

Thanks for integrating Codecov - We've got you covered ☂️