Add 2024 ATB Update Pipeline

[Yunzhi-Chen]

Summary

Adds a workflow that converts corrected ATB 2024 data into ReEDS-compatible inputs. It downloads, formats, plots, and validates ATB data, fills historical years from an explicitly configured source, optionally smooths future years, and produces before/after plots for review.

Structure

atb/
├── config.yaml            # all user choices; validated by config.schema.json
├── scripts/               # pipeline (13 modules)
├── historical/            # prepared history, committed — atb.csv, real.csv, manual.csv
├── manual_input/          # projection-year config (CSP ratios, offshore multipliers, coal overrides)
├── scraped_input/         # raw download cache (git-ignored)
├── output/                # generated ReEDS inputs (git-ignored)
└── comparison/            # validation and before/after plots (git-ignored)

How to run

cd atb
python scripts/future_atb_scraper.py     # cache the ATB flat file and workbook
python scripts/run_pipeline.py           # format, plot, compare

run_pipeline.py runs offline against the committed history and never rewrites it. Use --only format to generate CSVs without plots.

To rebuild history from original sources:

python scripts/historical_data_scraper.py    # --no-download to rebuild from cache

Logic

• Uses the ATB flat file for technology parameters and the workbook for battery power/energy costs. Preserves ReEDS filenames, schemas, and wind column ordering.
• Every metric selects its historical source explicitly, from four modes: atb (archived release-minus-two anchors, 41 selections), real (observed project costs, 13), manual (versioned ReEDS baseline, 4), and broadcast (first projection value held flat, 2).
• A metric can set one mode per sub-technology, so an observed series applies only to rows it actually measures — floating offshore, H-Frame, and aeroderivative variants get their own history rather than borrowing one.
• Observed history is scraped from LBNL land-based wind and utility-scale solar, the offshore wind market report, and EIA generator costs. Every row carries source file, sheet, dollar year, and SHA256.
• atb.csv holds only anchors, all satisfying year = ATB release − 2. The scraper verifies each workbook's own declared base year and raises on disagreement rather than assuming the offset.
• History is rebuilt from GitHub at the commit pinned in reeds_source.ref, so no local ReEDS clone is needed. A clone is still used for comparison baselines.
• Gap filling: interpolate between two known anchors, carry flat where only one side is known. This matters where ATB omits a technology until 2030 (nuclear, nuclear-SMR, floating offshore) — the curve rises to meet ATB instead of holding flat for years and then stepping.
• Smoothing bridges compact clusters of slope changes. Transitions of 10% or more, single slope changes, flat stretches, spans over four years, and adjustments under 0.5% are left as published. ATB's direction is never overridden. Applies to monetary metrics and CF multipliers only; heat rate and round-trip efficiency are never smoothed.
• Raw downloads, generated CSVs, and generated plots stay out of Git.

Verification

With smoothing disabled and all metrics set to manual, the workflow reproduces the inputs ReEDS uses today. Of 32 files: 21 exact, 8 differing only by floating-point noise (≤1e-9 relative), 3 offshore files differing by up to 0.018% in the CF multiplier, consistent with a different ATB 2024 release than the pinned v3.0.0.

Sanity-check plots — smoothing off, manual history, vs current ReEDS:

sanity_check_2024.zip

New ATB 2024 plots for future years:

New ATB 2024 plots and the components:

New ATB 2024 vs current ReEDS 2024:

plot_comparison_2024.zip

Scope and open questions

• Geothermal is not generated;
• The offshore fixed/floating multipliers (1.19 / 1.37) reproduce the current ReEDS file exactly but have no documented derivation in either repo. The 2025 file is currently a copy of the 2024 one.
• The offshore multiplier is applied to the projection and to atb-mode history.

LLM usage

LLM tools assisted with implementation, documentation, debugging, and validation. Outputs were reviewed and tested against the current ReEDS inputs.