Create reusable anonymized Project Ledger template

[rlancaster243]

What changed

• extracted the reusable Project Ledger architecture from DE-project-1
• replaced personal and environment-specific configuration with parameters and examples
• added a fixture-first SEC Company Facts ingestion pipeline
• added deterministic normalization, atomic local storage, and repeat-run tests
• added dbt/DuckDB staging and mart models
• added CI, Docker, Airflow, and parameterized GCP Terraform examples
• repositioned the README as a concise portfolio and onboarding entry point
• added a case study, engineering decision record, operational runbook, architecture guide, customization guide, and contribution guidance

Portfolio documentation

• README.md explains the project, architecture, demonstrated capabilities, scope, validation, boundaries, limitations, and next steps
• docs/CASE_STUDY.md records the engineering objective, implemented scope, failure modes, evidence, and limitations
• docs/DECISIONS.md explains the major architectural choices and trade-offs
• docs/OPERATIONS.md covers operating modes, monitoring, failure handling, replay, security, and deployment checks
• docs/ARCHITECTURE.md describes components and extension points
• docs/CUSTOMIZATION.md explains how another engineer can adapt the template

Deliberately excluded

• personal sprint journals and career-development notes
• generated dashboards, runtime logs, validation evidence, and historical incident artifacts
• real cloud project IDs, service accounts, buckets, email addresses, and secrets
• organization-specific conventions that would make the repository unusable elsewhere
• claims that this template is itself a currently hosted production service

Validation

The implementation validation covers:

• dependency installation with Python 3.12 and uv
• Ruff linting
• strict mypy type checking
• pytest test suite
• deterministic fixture-pipeline smoke test
• dbt/DuckDB build and data tests
• Terraform formatting
• Terraform initialization and validation

A repository-content scan found no personal names, employer references, private project IDs, or common credential patterns. No live SEC request or Terraform deployment was executed.

Impact

Another engineer can clone the repository, choose an issuer allowlist, configure an SEC user agent, run the fixture pipeline locally, build the DuckDB/dbt mart, understand the design decisions and operating expectations, and extend the same contracts toward Airflow and GCP.

Summary by CodeRabbit

• New Features
  • Added SEC company-facts ingestion in fixture and live modes.
  • Added normalization of financial observations into JSONL output, with raw payload preservation.
  • Added DuckDB/dbt staging and deduplicated financial fact models.
  • Added Docker, daily orchestration, and optional GCP deployment configuration.
  • Added configurable SEC access, retries, rate limits, and output locations.
• Documentation
  • Added setup, architecture, customization, operations, contribution, and engineering decision guides.
• Validation
  • Added automated linting, type checking, tests, dbt validation, and Terraform checks.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 0f15b65d-b916-4541-bce4-cb3a29d23623

📥 Commits

Reviewing files that changed from the base of the PR and between 57055e5 and 04bcdd1.

📒 Files selected for processing (34)

• .dockerignore
• .env.example
• .github/workflows/ci.yml
• .gitignore
• CONTRIBUTING.md
• Dockerfile
• Makefile
• README.md
• config/ledger.example.yml
• dags/ledger_daily_pipeline.py
• dbt/dbt_project.yml
• dbt/models/marts/fct_financial_facts.sql
• dbt/models/schema.yml
• dbt/models/staging/stg_financial_facts.sql
• dbt/profiles.yml
• docs/ARCHITECTURE.md
• docs/CASE_STUDY.md
• docs/CUSTOMIZATION.md
• docs/DECISIONS.md
• docs/OPERATIONS.md
• infra/gcp/main.tf
• infra/gcp/variables.tf
• pyproject.toml
• scripts/validate.sh
• src/ledger/__init__.py
• src/ledger/cli.py
• src/ledger/config.py
• src/ledger/pipeline.py
• src/ledger/sec_client.py
• src/ledger/storage.py
• tests/fixtures/CIK0000320193.json
• tests/fixtures/CIK0000789019.json
• tests/test_config.py
• tests/test_pipeline.py

---

📝 Walkthrough

Walkthrough

Project Ledger adds a configurable SEC/XBRL ingestion package with fixture and live clients, normalized JSONL outputs, DuckDB/dbt models, container and Airflow execution paths, GCP Terraform resources, CI validation, and accompanying architecture, operations, customization, and contribution documentation.

Changes

Project Ledger platform

Layer / File(s)	Summary
Ingestion and persistence core src/ledger/*, tests/*	Adds validated configuration, live and fixture SEC clients, company-facts normalization, atomic JSON/JSONL writes, CLI execution, public exports, fixtures, and repeatability/configuration tests.
Typed dbt warehouse models dbt/*	Reads normalized JSONL into typed staging views and builds a deduplicated financial-facts mart with schema metadata and data tests.
Runtime and cloud execution wiring Dockerfile, Makefile, config/*, dags/*, infra/gcp/*	Adds container and local command entrypoints, example runtime configuration, a daily Airflow DAG, and parameterized GCP resources for Cloud Run, storage, BigQuery, IAM, and Secret Manager.
Validation and repository tooling pyproject.toml, scripts/*, .github/workflows/*, .dockerignore, .gitignore	Configures packaging, development dependencies, Ruff, Mypy, pytest, dbt validation, CI checks, and generated-artifact exclusions.
Project documentation and operating guidance README.md, docs/*, CONTRIBUTING.md	Documents architecture, setup, validation, operating procedures, engineering decisions, customization, limitations, and contribution requirements.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Operator
  participant ledger.cli
  participant CompanyFactsClient
  participant ledger.pipeline
  participant DuckDB/dbt
  Operator->>ledger.cli: Run fixture or live command
  ledger.cli->>CompanyFactsClient: Fetch company facts
  CompanyFactsClient-->>ledger.cli: Return validated payload
  ledger.cli->>ledger.pipeline: Normalize and persist outputs
  ledger.pipeline-->>ledger.cli: Return pipeline summary
  DuckDB/dbt->>ledger.pipeline: Read normalized JSONL
  DuckDB/dbt-->>Operator: Build staging views and financial mart

Loading

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/reusable-ledger-template

---

_{Comment @coderabbitai help to get the list of available commands.}