Keep an escaped value that equals the null string

[saleemno1]

CSVParser.handleNull compares the fully unescaped token against the format's null string, so a value that really is \N and that the printer correctly wrote as \\N for MYSQL, POSTGRESQL_TEXT, POSTGRESQL_CSV and ORACLE comes back as null; the lexer now records whether an escape sequence was actually translated, and only an untranslated token can be the null marker.

• Read the contribution guidelines for this project.
• Read the ASF Generative Tooling Guidance if you use Artificial Intelligence (AI).
• I used AI to create any part of, or all of, this pull request. Which AI tool was used to create this pull request, and to what extent did it contribute?
• Run a successful build using the default Maven goal with mvn; that's mvn on the command line by itself.
• Write unit tests that match behavioral changes, where the tests fail if the changes to the runtime are not applied. This may not always be possible, but it is a best practice.
• Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
• Each commit in the pull request should have a meaningful subject line and body. Note that a maintainer may squash commits during the merge process.

[garydgregory]

Hello @saleemno1
Please see my comment.

[garydgregory]

Use a JUnit @ParameterizedTest instead of a loop.

[saleemno1]

Done, it's an @ParameterizedTest with @EnumSource now. I also folded PostgreSQLCsv into the list since it has the same \N null string and was missing from the loop.

[garydgregory]

Hello @saleemno1
Thank you for your update. Please see my comment.

[garydgregory]

Refactor the magic string \\N into a constant or a local variable with a name that explains its intent. Since there are 2 such instances in this method, it will also make it obvious that both magic strings are indeed intended to be the same.

[garydgregory]

@saleemno1 Ping on #626 (comment) 👋

[Copilot]

Pull request overview

Fixes a null-marker ambiguity when parsing formats that use a null string (for example \N): if a literal value equal to the null string was emitted with escape processing (e.g., \\N on output), the parser previously unescaped it back to \N and incorrectly returned null. The lexer now tracks whether any escape sequence was actually translated for the current token, and the parser only treats an untranslated nullString token as a null marker.

Changes:

• Add Token.isEscaped and set it in the lexer when escape handling translates input.
• Update CSVParser.handleNull() to preserve values equal to nullString when the token involved escape translation.
• Add/adjust unit tests to cover escaped-null-string round-tripping across relevant predefined formats.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/main/java/org/apache/commons/csv/Token.java	Adds a per-token flag to record whether escape translation occurred while building token content.
src/main/java/org/apache/commons/csv/Lexer.java	Sets the new flag when an escape sequence is translated into token content.
src/main/java/org/apache/commons/csv/CSVParser.java	Uses the new lexer signal to avoid treating escaped literals as null markers.
src/test/java/org/apache/commons/csv/CSVParserTest.java	Adds a parameterized regression test for escaped nullString values.
src/test/java/org/apache/commons/csv/CSVPrinterTest.java	Updates random round-trip expectations to account for when nullString is printed verbatim vs escaped/quoted.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

src/test/java/org/apache/commons/csv/CSVPrinterTest.java:168

• expectNulls currently assumes that any non-verbatim rendering of the null string means the parser will read the value back as a literal. This is not true when QuoteMode.ALL is used: both null and a literal value equal to nullString are printed quoted, so they are indistinguishable and the parser will still produce null. Consider treating QuoteMode.ALL as indistinguishable from verbatim printing for this helper.

        final String nullString = csvFormat.getNullString();
        if (nullString == null || !printsVerbatim(csvFormat, nullString)) {
            return fixed;
        }

src/test/java/org/apache/commons/csv/CSVPrinterTest.java:161

• This JavaDoc implies that quoting the null string is sufficient for the parser to read it back as a literal value. In this codebase, quoted null markers are only distinguished in strict quote modes (ALL_NON_NULL / NON_NUMERIC); otherwise a quoted value equal to the null string is still parsed as null.

     * Converts an input CSV array into expected output values, including NULLs. A value equal to the null string is converted to null only when the printer
     * writes that value unchanged; when the printer escapes or quotes it, the parser reads it back as the value it is.

[garydgregory]

@saleemno1
Please review the copilot comments and at least improve the code comments.
TY!

[garydgregory]

@saleemno1 ping 🔔