fix: zero-pad years below 1000 in date/datetime formatting

[Danish-sarwar1]

Problem

Source MySQL tables contain typo'd temporal values with years < 1000 (e.g. dob = '0979-02-15', active_start_date = '0202-04-17'). Debezium emits these faithfully as epoch-days/millis/micros, but the batcher's formatters print the year with %d (unpadded), so the S3 JSON carries "202-04-17" — 9 characters.

Redshift COPY … json 'auto' rejects such values on date/timestamp columns:

err 1205: Invalid Date Format - length must be 10 or more

The loader then aborts and retries the same batch forever — the sink stalls and Redshift burns cycles re-running the same COPY every ~40s.

This stalled 9 sinks in production (availability, contacts ×2, payments, posttreatment ×2, soap_notes ×2) for 20–30+ hours each during the new-pipeline reloads. The bug has been latent since 2020 (01d02bb); day-to-day streaming rarely carries such rows, but full re-snapshots push every historical row through this path.

Fix

%04d for the year in convertDebeziumDate, convertDebeziumMilliseconds, convertDebeziumMicroseconds. Years 1000–9999 format byte-identically to before (the pre-existing test cases pass unchanged); years < 1000 now serialize as valid YYYY-MM-DD and load into Redshift faithfully (0202-04-17 is a valid Redshift date).

Tests

Extended TestConvertDebeziumFormattedTime:

• normal-date regression coverage: epoch (1970-01-01), recent (2022-07-08), max (9999-12-31) — plus the pre-existing 1934/1988/2020 cases, all unchanged
• boundary pair: 0999-12-31 / 1000-01-01
• incident reproductions: DATE 0202-04-17, 0979-02-15; DATETIME (ms path) 0204-10-15 00:00:00; DATETIME(6) (µs path) 0201-08-01 10:30:00.757000

gofmt, go vet, and the full go test ./cmd/... ./pkg/... suite pass in the CI image (public.ecr.aws/practo/golang:1.17.1-alpine).

Note on already-produced batches

This fixes files the batcher writes after rollout. Batches already uploaded to S3 with unpadded years still fail COPY — currently being drained by padding them in place in S3; adding ACCEPTANYDATE to the loader COPY options is a possible follow-up safety net.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes

  • Corrected Debezium date and timestamp formatting to consistently use four-digit, zero-padded years.
  • Improved handling of dates and timestamps from years before 1000.

• Tests

  • Added coverage for epoch dates, recent dates, maximum supported dates, and early-year timestamps.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8964f92b-7487-4814-87b4-7102bc3f2361

📥 Commits

Reviewing files that changed from the base of the PR and between 312e224 and e930747.

📒 Files selected for processing (2)

• pkg/transformer/debezium/message.go
• pkg/transformer/debezium/message_test.go

---

Walkthrough

Changes

Debezium temporal formatting

Layer / File(s)	Summary
Zero-padded temporal years pkg/transformer/debezium/message.go, pkg/transformer/debezium/message_test.go	Date and timestamp conversions now emit four-digit years, with tests covering epoch, boundary dates, and years below 1000.

Estimated code review effort: 1 (Trivial) | ~5 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: zero-padding years below 1000 in date and datetime formatting.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/zero-pad-temporal-years

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}