Skip to content

csv-core: make is_non_numeric cheaper - #442

Open
patrick-immometrica wants to merge 1 commit into
BurntSushi:masterfrom
patrick-immometrica:perf-non-numeric
Open

patrick-immometrica wants to merge 1 commit into
BurntSushi:masterfrom
patrick-immometrica:perf-non-numeric

Conversation

@patrick-immometrica

Copy link
Copy Markdown

With QuoteStyle::NonNumeric, every field goes through parse::<f64>() and then parse::<i128>(). Text fields fail both, so they pay for two parses each.

Changes:

  • fields that are only digits (with an optional sign) return without parsing
  • fields whose first byte can't start a number return right away
  • the i128 parse is gone, since f64 accepts every string i128 does

Writer benchmarks with NonNumeric (time per iteration):

data master this PR
nfl 1.60 ms ~1.00 ms
game 6.24 ms ~3.7 ms

For reference, Necessary is ~0.8 ms and ~2.3 ms on the same data.

I checked the new function against the old one on about 1M generated inputs (signs, exponents, inf/nan spellings, junk) and got the same result for all of them. Also added a small unit test.

`QuoteStyle::NonNumeric` ran `parse::<f64>()` and then `parse::<i128>()`
on every field, so text fields always paid for two failed parses.

Plain integers are now detected with a byte scan, fields whose first byte
can't start a number return early, and the `i128` parse is dropped since
any input it accepts is also accepted by the `f64` parser.

Writing the nfl and game benchmark data with `NonNumeric` drops from
1.60 ms to ~1.00 ms and from 6.24 ms to ~3.7 ms. I compared the old and
new function on about a million generated inputs with no difference.
@patrick-immometrica
patrick-immometrica marked this pull request as ready for review November 14, 2026 11:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant