Skip to content

format: a CSV can be written in the dialect you were handed #36

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
donislawdev merged 1 commit into from
Sep 1, 2026
Merged

format: a CSV can be written in the dialect you were handed #36

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,31 @@ because it turns other people's test suites red.

### Added

- **A CSV can be written in the dialect you were handed.** `--set
delimiter=semicolon`, `--set line_ending=crlf` and `--set header=false` on a
`csv`, separately or together. Separators are named rather than typed, so
`tab` and `pipe` need no escaping: the four are `comma`, `semicolon`, `tab`
and `pipe`.

These are the three ways a real CSV differs before its contents do. A
European spreadsheet exports with semicolons, anything written on Windows
ends its rows with CRLF, and a table dumped straight out of a database has no
header. All three are CSV and all three break a reader that assumed the other
thing.

The description column keeps carrying the separator, so a semicolon file
still exercises quoted fields rather than quietly testing less than a comma
one does.

Two things worth knowing. The smallest file changes with the dialect, because
a CRLF row is a byte longer and a header is a whole line - the tool tells you
the floor for the settings you gave it. And the manifest records the
separator as the character that is in the file, where the recipe names it as
a word.

The defaults are `comma`, `lf` and a header, which is what this tool has
always written, so **no existing file changes by a byte**.

- **A log can be made quiet, or full of errors.** `--set level_mix=errors` on
a `log`, with `realistic`, `quiet`, `errors` and `debug` to choose from. It
decides which severities appear, the way `status_mix` already decides which
Expand Down
168 changes: 114 additions & 54 deletions internal/format/csvfile/csv.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,17 +35,16 @@ import (
const (
generatorVersion = "1"

header = "id,name,email,amount,created,description\n"
emailDomain = "@example.com"
createdDate = "2026-08-01"

// amountWidth is six digits, a dot and two more. The range the amount is
// drawn from below guarantees it.
amountWidth = 9

// rowTail is what follows the description: the closing quote and the
// newline.
rowTail = `"` + "\n"
// closingQuote ends the description. What follows it is the row ending,
// which the dialect decides, so the two are no longer one constant.
closingQuote = `"`

// maxRowDigits bounds the width of the row number. A row is at least one
// byte, so a file can never hold more rows than it has bytes, and a size is
Expand All @@ -54,13 +53,24 @@ const (
// length whatever row number it lands on.
maxRowDigits = 19

// fixedWidth is every byte of a row except the row number, the name (which
// also forms the address) and the description. A constant expression, so it
// cannot drift away from the template above.
fixedWidth = 5 /* separators */ + len(emailDomain) + amountWidth +
len(createdDate) + 1 /* the opening quote */ + len(rowTail)
// fixedBeforeEnding is every byte of a row except the row number, the name
// (which also forms the address), the description and the row ending. A
// constant expression, so it cannot drift away from the template above.
//
// The five separators count one byte each, which is a fact about the
// separators offered rather than an assumption: every one of them is a
// single byte, and dialect.go says so where they are declared.
fixedBeforeEnding = 5 /* separators */ + len(emailDomain) + amountWidth +
len(createdDate) + 1 /* the opening quote */ + len(closingQuote)
)

// fixedWidth is fixedBeforeEnding plus the row ending, which the dialect
// decides. A CRLF row costs one byte more than an LF one, on every row, which
// is why the minimum moves with this setting.
func fixedWidth(d dialect) int64 {
return int64(fixedBeforeEnding + len(d.eol))
}

func init() {
format.Register(format.Descriptor{
ID: "csv",
Expand All @@ -70,9 +80,14 @@ func init() {

// A file with a header and no rows is legal CSV, and it is not something
// anybody orders by naming a byte count - that is a shape request, and
// it arrives with the row count property. The minimum here is the header
// and one whole row.
MinBytes: minimumBytes(),
// it would arrive with a row count setting, which this format does not
// offer yet.
//
// The floor announced here is for the settings left alone. The dialect
// moves the real one - a CRLF row costs a byte more and a file with no
// header has one fewer line to pay for - so Plan works that one out and
// names it. The log format is arranged the same way.
MinBytes: minimumBytes(defaultDialect()),

Padding: format.PaddingChannel{
Name: "the description field of the last row",
Expand All @@ -85,26 +100,34 @@ func init() {
// name and the manifest carry it instead.
Label: format.LabelExternalOnly,
Oracle: "python-csv",
// Separator, quoting, column count and column types come later.
// Declaring none now makes a recipe asking for them fail loudly.
Properties: nil,
// Quoting, column count and column types come later. Declaring none of
// them now makes a recipe asking for one fail loudly.
Properties: properties(),
GeneratorVersion: generatorVersion,
Generator: generator{},
})
}

type generator struct{}

type memo struct{ seed uint64 }
type memo struct {
seed uint64
dia dialect
}

func (generator) Plan(r format.Request) (format.Plan, error) {
min := minimumBytes()
d, err := parseDialect(r.Properties)
if err != nil {
return format.Plan{}, err
}

min := minimumBytes(d)
if r.Bytes < min {
return format.Plan{}, &format.BelowMinimumError{
Format: "CSV",
Requested: r.Bytes,
Minimum: min,
Reason: "a table holds a header and whole rows, and one of each needs that much",
Reason: reasonForMinimum(d),
Hint: fmt.Sprintf("Ask for %d B or more.", min),
}
}
Expand All @@ -114,42 +137,62 @@ func (generator) Plan(r format.Request) (format.Plan, error) {
Exact: true,
Determinism: format.DeterminismByte,
Properties: map[string]any{
"encoding": "utf-8",
"line_ending": "lf",
"separator": ",",
"header": true,
"columns": 6,
"encoding": "utf-8",
// The manifest carries the separator as the CHARACTER, where the
// recipe names it as a word. That difference is deliberate and is
// the same one the contract already draws between size, which is an
// intention, and bytes, which is a fact. Changing it would break
// every script reading this field.
"line_ending": d.lineEndingID,
"separator": string(d.sep),
"header": d.header,
"columns": len(columnNames),
// Stated even though it is always false here, so a test can assert
// on it without knowing which formats carry a label internally.
format.PropertyLabelEmbedded: false,
},
Memo: memo{seed: r.Seed},
Memo: memo{seed: r.Seed, dia: d},
}, nil
}

// reasonForMinimum says what the floor is made of, which changes with the
// dialect. A file with no header pays for rows alone, and saying "a header and
// whole rows" there would name something the file does not have.
func reasonForMinimum(d dialect) string {
if !d.header {
return "a table holds whole rows, and one of them needs that much"
}
return "a table holds a header and whole rows, and one of each needs that much"
}

func (generator) Write(ctx context.Context, w io.Writer, p format.Plan) error {
m, ok := p.Memo.(memo)
if !ok {
return fmt.Errorf("csv: the plan was not produced by this generator")
}

if err := core.WriteAll(w, []byte(header)); err != nil {
return err
if m.dia.header {
if err := core.WriteAll(w, []byte(m.dia.headerLine())); err != nil {
return err
}
}

rng := core.NewRand(m.seed)
return core.FillRecords(ctx, w, rng, p.Bytes-int64(len(header)), &rows{})
return core.FillRecords(ctx, w, rng, p.Bytes-m.dia.headerBytes(), &rows{dia: m.dia})
}

// rows builds the data rows. It carries the row number, so the id column counts
// up the way a real export does.
type rows struct{ next int64 }
type rows struct {
next int64
dia dialect
}

// Shortest is the smallest row this builder can close a file with: the widest
// row number, the longest name in both the name and the address, and an empty
// description. It has to hold for every draw rather than for the lucky one.
func (r *rows) Shortest() int64 {
return int64(maxRowDigits + 2*longestWord + fixedWidth)
return int64(maxRowDigits+2*longestWord) + fixedWidth(r.dia)
}

func (r *rows) Append(dst []byte, rng *rand.Rand) []byte {
Expand Down Expand Up @@ -185,42 +228,51 @@ func (r *rows) append(dst []byte, rng *rand.Rand, want int64) []byte {
whole := 100000 + rng.IntN(899999)
cents := rng.IntN(100)

sep := r.dia.sep

dst = strconv.AppendInt(dst, r.next, 10)
dst = append(dst, ',')
dst = append(dst, sep)
dst = append(dst, name...)
dst = append(dst, ',')
dst = append(dst, sep)
dst = append(dst, name...)
dst = append(dst, emailDomain...)
dst = append(dst, ',')
dst = append(dst, sep)
dst = strconv.AppendInt(dst, int64(whole), 10)
dst = append(dst, '.')
if cents < 10 {
dst = append(dst, '0')
}
dst = strconv.AppendInt(dst, int64(cents), 10)
dst = append(dst, ',')
dst = append(dst, sep)
dst = append(dst, createdDate...)
dst = append(dst, ',', '"')
dst = append(dst, sep, '"')

if want < 0 {
dst = appendPhrase(dst, rng, 3+rng.IntN(5))
dst = appendPhrase(dst, rng, 3+rng.IntN(5), sep)
} else {
// Everything written so far, plus what still has to follow.
used := int64(len(dst)-start) + int64(len(rowTail))
dst = appendFiller(dst, want-used)
used := int64(len(dst)-start) + int64(len(closingQuote)) + int64(len(r.dia.eol))
dst = appendFiller(dst, want-used, sep)
}

return append(dst, rowTail...)
dst = append(dst, closingQuote...)
return append(dst, r.dia.eol...)
}

// appendPhrase writes a readable description. Every few words it drops a comma,
// which is the case a CSV reader has to get right and the reason the column is
// quoted at all.
func appendPhrase(dst []byte, rng *rand.Rand, n int) []byte {
// appendPhrase writes a readable description. Every few words it drops the
// SEPARATOR, which is the case a CSV reader has to get right and the reason the
// column is quoted at all.
//
// The separator rather than always a comma, and that is the point of the
// setting rather than a detail of it. A comma inside a semicolon separated file
// needs no quoting, so a description that kept dropping commas would leave a
// semicolon file never exercising the quoted path at all - the file would be
// the right size, parse everywhere, and quietly test less than the comma one.
func appendPhrase(dst []byte, rng *rand.Rand, n int, sep byte) []byte {
for i := 0; i < n; i++ {
if i > 0 {
if i%3 == 0 {
dst = append(dst, ',')
dst = append(dst, sep)
}
dst = append(dst, ' ')
}
Expand All @@ -236,24 +288,32 @@ func appendPhrase(dst []byte, rng *rand.Rand, n int) []byte {
// field early.
// appendFiller stretches the description to the byte.
//
// A comma every fourth word, unlike every other format here, and on purpose:
// the description is a quoted field, so the padding is what makes a long file
// keep exercising the quoting rather than turning into plain words.
func appendFiller(dst []byte, n int64) []byte {
// A separator every fourth word, unlike every other format here, and on
// purpose: the description is a quoted field, so the padding is what makes a
// long file keep exercising the quoting rather than turning into plain words.
// It follows the dialect for the reason appendPhrase gives.
func appendFiller(dst []byte, n int64, sep byte) []byte {
both := string(sep) + " "
return core.AppendFiller(dst, words, n, func(i int) string {
if i%4 == 0 {
return ", "
return both
}
return " "
})
}

// minimumBytes is the header and one whole row, computed rather than written
// down so it cannot drift away from the template the way a number in a document
// would.
func minimumBytes() int64 {
var r rows
return int64(len(header)) + r.Shortest()
// minimumBytes is the header, when there is one, and one whole row. Computed
// rather than written down so it cannot drift away from the template the way a
// number in a document would.
//
// It takes the dialect because the floor moves with it: a CRLF row costs a byte
// more, and a file with no header has one fewer line to pay for. The registry
// announces the floor for the settings left alone, and Plan works out the real
// one for the settings that arrived - the same arrangement the log format uses,
// where the entry shape moves the floor too.
func minimumBytes(d dialect) int64 {
r := rows{dia: d}
return d.headerBytes() + r.Shortest()
}

// longestWord is the widest draw, because the minimum has to hold for every
Expand Down
Loading
Loading