csv-preflight/validate-csv-github-actions.html at main · softpeanut/csv-preflight · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width,initial-scale=1">
  <meta name="description" content="Validate CSV structure in GitHub Actions, keep files on the runner, preserve issue reports after failure, and avoid silently repairing ambiguous data.">
  <link rel="canonical" href="https://softpeanut.github.io/csv-preflight/validate-csv-github-actions.html">
  <title>How to Validate CSV Files in GitHub Actions</title>
  <link rel="icon" href="logo.svg" type="image/svg+xml">
  <link rel="stylesheet" href="styles.css">
  <style>
    .article{max-width:780px;margin:0 auto;padding:3rem 1.25rem 5rem}.article h1{font-size:clamp(2rem,7vw,3.7rem);line-height:1.04}.article h2{margin-top:2.5rem}.article p,.article li{line-height:1.75}.article pre{overflow-x:auto;padding:1rem;border:1px solid #d7dce5;border-radius:.6rem;background:#f7f8fa}.article code{font-family:ui-monospace,SFMono-Regular,Menlo,monospace}.article .lede{font-size:1.2rem;color:#495164}.article .note{padding:1rem 1.2rem;border-left:4px solid #3659d9;background:#eef2ff}.article nav{margin-bottom:2rem}
  </style>
</head>
<body>
<main class="article">
  <nav><a href="./">Try CSV Preflight</a> · <a href="https://github.com/softpeanut/csv-preflight-action/releases/tag/v1.0.0">Action release</a> · <a href="pro-terms.html">Pro terms</a></nav>
  <article>
    <h1>Validate CSV Files in GitHub Actions</h1>
    <p class="lede">Fail a workflow on broken structure, retain the diagnostic files, and keep the CSV on the runner.</p>

    <p>A CSV import can fail long after review if a header is duplicated, one row has an extra column, or an export silently changes encoding. A CI check catches those structural problems next to the commit that introduced them. It should not pretend to know which duplicate is correct or rewrite business values.</p>

    <h2>A complete workflow</h2>

    <p>This workflow checks one UTF-8 file. The preflight step exits with status 1 when it finds issues, so the job fails. The artifact step uses <code>if: always()</code>, which preserves the normalized CSV and issue report even after that expected failure.</p>

    <pre><code>name: Validate import CSV

on:
  pull_request:
    paths:
      - "data/import.csv"
  push:
    branches: [main]
    paths:
      - "data/import.csv"

permissions:
  contents: read

jobs:
  csv-preflight:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4

      - name: Check CSV structure
        uses: softpeanut/csv-preflight-action@eb04c527a46ce3bc8bfc711fde8e93ca947597ae
        with:
          path: data/import.csv
          normalized_path: ${{ runner.temp }}/import.normalized.csv
          report_path: ${{ runner.temp }}/import.issues.csv

      - name: Keep preflight artifacts
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: csv-preflight
          path: |
            ${{ runner.temp }}/import.normalized.csv
            ${{ runner.temp }}/import.issues.csv
          if-no-files-found: warn</code></pre>

    <p>The full commit SHA pins the checked v1.0.0 Action bytes. A shorter <code>@v1</code> reference is easier to read, but a full SHA prevents a tag move from changing third-party code without a workflow diff. Apply the same pinning policy to every external Action when your threat model requires immutable dependencies.</p>

    <h2>What the Action checks</h2>

    <ul>
      <li>UTF-8, UTF-8 BOM, unsupported UTF-16 markers, and invalid UTF-8;</li>
      <li>comma, tab, semicolon, or pipe delimiter inference;</li>
      <li>quoted delimiters, escaped quotes, and quoted newlines;</li>
      <li>empty or duplicate headers;</li>
      <li>rows whose column count differs from the header;</li>
      <li>exact duplicate data rows.</li>
    </ul>

    <p>It preserves data rows. Duplicate rows and uneven rows are reported, not deleted, padded, or truncated. Empty and repeated header names are made unique only in the normalized output. The issue report records the detected type, row, and detail.</p>

    <div class="note"><strong>Exit codes:</strong> 0 means the declared structural checks found no issue; 1 means issues were reported or the input was rejected; 2 means invocation or runtime failure. Exit 0 is not a guarantee that an importer accepts the file.</div>

    <h2>Why the artifact step needs <code>always()</code></h2>

    <p>GitHub normally skips later steps after a failure. That would hide the report at the moment it is most useful. <code>if: always()</code> lets the artifact upload run while leaving the original preflight step—and therefore the job—failed. Reviewers get both a red check and the evidence needed to fix it.</p>

    <p>For an encoding or syntax rejection, a normalized file may not exist. <code>if-no-files-found: warn</code> allows the report to upload without turning that expected absence into a second error. Do not use <code>continue-on-error</code> on the preflight step unless the workflow is intentionally advisory; doing so converts data-quality failures into green jobs.</p>

    <h2>Keep the permissions narrow</h2>

    <p>The Action is a composite wrapper around a dependency-free Node.js CLI. It reads the checked-out path and writes only to the paths supplied by the workflow. It does not upload the CSV or require an API key. The workflow grants <code>contents: read</code>, which is enough for checkout.</p>

    <p>“No upload” is a data-flow statement, not a universal security guarantee. The runner and any later artifact step are still part of your trust boundary. Do not place secrets or regulated data in a repository or artifact merely because the validator itself has no network call. Configure artifact retention and repository access for the sensitivity of the file.</p>

    <h2>Scope the trigger to the file</h2>

    <p>The <code>paths</code> filter avoids running the job on unrelated changes. For generated CSV, include the generator source paths as well so a code change that alters output can exercise the check. If several files must be validated, a matrix can call the free Action once per file, but it does not produce a combined manifest.</p>

    <p>The free boundary is one generic file up to 10 MiB per invocation. The separate Pro edition processes up to 20 files in one run and adds named import profiles, one ZIP, a manifest, and JSON output. It is optional; the workflow above is complete and usable without buying anything.</p>

    <h2>Reproduce a failure locally</h2>

    <p>Download <a href="cli.mjs">the same one-file CLI</a> and run it with Node.js 20 or newer:</p>

    <pre><code>node cli.mjs data/import.csv \
  --output /tmp/import.normalized.csv \
  --report /tmp/import.issues.csv
echo $?</code></pre>

    <p>The local command and Action share the same implementation and exit-code contract. Existing output paths are never overwritten, which prevents a stale report from being mistaken for the current run. Remove or choose new output paths deliberately before rerunning.</p>

    <h2>Use schema checks separately</h2>

    <p>Structural validation cannot decide whether a price is negative, a customer ID exists, or a date belongs to the required timezone. Add importer-specific schema and business-rule checks after structural parsing. Keep their reports separate so reviewers can distinguish malformed CSV from valid CSV containing invalid domain values.</p>

    <p>Start with the <a href="./">browser checker</a>, inspect the <a href="https://github.com/softpeanut/csv-preflight-action">standalone Action source and tests</a>, review the <a href="csv-ci-case-study.html">public 1,583-row reproduction</a>, or use the public <a href="https://github.com/softpeanut/csv-preflight-action/releases/tag/v1.0.0">v1.0.0 Action release</a>. For offline batch and team use, the <a href="pro-terms.html">published Pro terms</a> describe the completed product. If you want this exact one-file workflow configured against a public or sanitized repository, the separate <a href="ci-setup-terms.html">$99 fixed-scope setup terms</a> define every deliverable and exclusion before payment.</p>
  </article>
</main>
</body>
</html>