arrow::csv silently drops one byte when a CRLF inside a quoted field straddles a parser block boundary

[hanke580]

Describe the bug, including details regarding any error messages, version, and platform.

arrow::csv silently drops one byte when a CRLF inside a quoted field straddles a parser
block boundary. The \r is kept and the \n disappears. No error, no warning — the value
simply comes back one byte shorter than it was written.

This reproduces on Arrow C++ main, in C++, with no Python in the picture, and at the
shipped default block size with no options tuned.

Build under test

Arrow C++ main @ 0bd8def  ("GH-51343: [C++][CI][Packaging] Disable Precompile Headers
                            on simdjson...", 2026-09-17)
libarrow.so.2600.0.0, 26.0.0-SNAPSHOT, Release build
cmake 4.4.3 / g++ 11.4.0 / Ubuntu 22.04 x86_64

Also reproduces on released pyarrow 25.0.1 (newest on PyPI), on all three entry points —
csv.read_csv, the streaming csv.open_csv, and the dataset API — which is what places it
in the shared reader rather than in a wrapper.

Minimal case. A CSV whose row 0 holds a quoted value containing one embedded CRLF.
newlines_in_values = true is required for such a file; without it the reader rejects the
file outright (Expected 3 columns, got 2) rather than corrupting it.

id,s,p
0,"zzzz…zzzA<CR><LF>B",x
1,plain,x
2,plain,x

Walking that CRLF one byte at a time across a 1024-byte block boundary, 1028 offsets tried
on main:

block_size = 1024, walking one embedded CRLF across the boundary

  pad=1012: CR at byte 1023, LF at byte 1024, boundary at 1024
    wrote 1016B ...A\r\n   read 1015B ...zA\r

1 corrupting offset(s), 0 error(s), out of 1028 tried.

Exactly one offset out of 1028: the one where the CR is the last byte of a block and the
LF is the first byte of the next.

It is not an artefact of a small block size. With no ReadOptions at all, so the shipped
1 MiB block applies, on main:

 file bytes  CR at byte  verdict
    1048602     1048575  CORRUPT: LF silently dropped
    1048603     1048576  ok
    1048604     1048577  ok
    1048605     1048578  ok
    1048606     1048579  ok

It is specific to the CR–LF pair. Same sweep on main, varying the byte sequence placed
inside the quoted field (1032 offsets each):

sequence inside the quoted field	corrupting offsets
A\r\nB (embedded CRLF)	1
A\r\r\nB (CR CR LF)	1
A\nB (lone LF)	0
A\rB (lone CR)	0
A\n\rB (LF then CR)	0
A€B (3-byte UTF-8)	0

A lone \n or a lone \r at exactly the same boundary survives; only the two sequences that
contain a \r immediately followed by \n lose a byte, and the CR-CR-LF case is the same
defect one byte over. So this is not general boundary mishandling — the doubled-quote escape
and multi-byte UTF-8 paths are sound across the boundary.

Reproduction

C++ (this is the one that runs against main). Build Arrow with -DARROW_CSV=ON, then:

#include <arrow/csv/api.h>
#include <arrow/io/api.h>
#include <arrow/table.h>
#include <arrow/array.h>
#include <fstream>
#include <cstdio>

int main() {
  const std::string PREFIX = "id,s,p\n0,\"", VAL = "A\r\nB", PATH = "/tmp/f.csv";
  const int block = 1024;
  for (int pad = block - 1020; pad < block + 8; pad++) {
    { std::ofstream f(PATH, std::ios::binary);
      f << PREFIX << std::string(pad, 'z') << VAL << "\",x\n1,plain,x\n2,plain,x\n"; }
    auto file = arrow::io::ReadableFile::Open(PATH).ValueOrDie();
    auto ro = arrow::csv::ReadOptions::Defaults();  ro.block_size = block;
    auto po = arrow::csv::ParseOptions::Defaults(); po.newlines_in_values = true;
    auto co = arrow::csv::ConvertOptions::Defaults();
    co.column_types = {{"id", arrow::int32()}, {"s", arrow::utf8()}, {"p", arrow::utf8()}};
    auto table = arrow::csv::TableReader::Make(arrow::io::default_io_context(), file,
                                               ro, po, co).ValueOrDie()->Read().ValueOrDie();
    auto col = table->GetColumnByName("s");
    std::string got;
    for (int i = 0; i < col->num_chunks(); i++) {
      auto a = std::static_pointer_cast<arrow::StringArray>(col->chunk(i));
      if (a->length() > 0) { got = a->GetString(0); break; }
    }
    std::string want = std::string(pad, 'z') + VAL;
    if (got != want)
      printf("pad=%d: wrote %zuB, read %zuB -> LF dropped\n", pad, want.size(), got.size());
  }
}

Prints exactly one line, at pad=1012. Drop ro.block_size = block and use
pad = (1<<20) - PREFIX.size() - 2 to see the same at the shipped default.

Python (released pyarrow 25.0.1), for convenience:

import pyarrow as pa, pyarrow.csv as pc

PREFIX, VAL = b'id,s,p\n0,"', "A\r\nB"

def check(pad, block):
    with open("/tmp/f.csv", "wb") as fh:
        fh.write(PREFIX + b"z" * pad + VAL.encode() + b'",x\n1,plain,x\n2,plain,x\n')
    t = pc.read_csv("/tmp/f.csv",
                    read_options=pc.ReadOptions(block_size=block) if block else pc.ReadOptions(),
                    parse_options=pc.ParseOptions(newlines_in_values=True),
                    convert_options=pc.ConvertOptions(
                        column_types={"id": pa.int32(), "s": pa.string(), "p": pa.string()}))
    return t.column("s").to_pylist()[0] == "z" * pad + VAL

print(check(1012, 1024))            # False  <- LF dropped
print(check((1 << 20) - len(PREFIX) - 2, None))   # False, at the shipped 1 MiB block

polars and duckdb read the identical bytes back correctly, so the file is well formed and
this is the reader, not the writer.

Where it comes from (a lead, not a confirmed diagnosis)

I have not bisected this to a single line, but the shape of the failure matches a specific
pattern that appears in both the chunker and the parser: a \r handler that looks ahead one
byte for the \n and gives up if the byte is not in the current buffer.

cpp/src/arrow/csv/chunker.cc, in the lexer's unquoted-field path:

if (ARROW_PREDICT_FALSE(c == '\r')) {
  if (ARROW_PREDICT_TRUE(data != data_end) && *data == '\n') {
    data++;
  }
  goto LineEnd;      // CR at the end of the buffer => treated as a complete terminator
}

cpp/src/arrow/csv/parser.cc has the same look-ahead guard (data < data_end && *data == '\n'), and additionally treats a \n that is the first byte it sees as an empty line:

// Special case empty lines: do we start with a newline separator?
c = *data;
if (ARROW_PREDICT_FALSE(IsControlChar(c))) {
  ...
  if (c == '\n') {
    data++;
    goto EmptyLine;
  }
}

A CR that ends a buffer is therefore accepted as a whole line terminator, and the orphaned LF
that opens the next buffer is then consumed as a separator rather than as data. That is
consistent with every observation above: it needs the pair, it needs the split to fall
between the two bytes, and a lone CR or lone LF at the same offset is unaffected.

FindFirstInternal in chunker.cc runs the lexer over partial and then over block
(cpp/src/arrow/csv/chunker.cc, LexingBoundaryFinder), which is the seam where the
carried-over state would need to remember "the previous buffer ended on a CR" for the pair to
be reassembled.