Skip to content

Fix log file errors - #211

Merged
wesleyjcole merged 6 commits into
mainfrom
wjc/log_updates
Sep 14, 2026
Merged

wesleyjcole merged 6 commits into
mainfrom
wjc/log_updates

Conversation

@wesleyjcole

@wesleyjcole wesleyjcole commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

A typical gamslog.txt file has many ERROR entries (my USA_default case from main had 554 entries). These have sometimes confused users, especially because these runs have completed successfully. This pull request cleans up most of the ERROR and warning noise in gamslog.txt, and fixes the pandas 3 Copy-on-Write (CoW) bugs that were associated with some of those messages. The remaining issues are primarily from reeds_to_rev.py which is currently not funtional.

Technical details

Implementation notes

Logging (reeds/log.py)makelog() redirects sys.stderr into the logger at logging.ERROR. Python's warnings module writes to stderr, so every warning was labeled ERROR. Adding logging.captureWarnings(True) sends warnings.warn() through the logger at WARNING instead.

Chained assignment (retail rate module, reeds_to_rev.py) — In df['col'].fillna(..., inplace=True), df['col'] returns a temporary Series and the mutation is applied to that temporary Series, which is then discarded. Pre-CoW, df['col'] was often a "view", so the write happened to propagate. pandas 3 makes CoW unconditional, so it no longer propogates. Specific issues:

  • postprocessing/retail_rate_module/ferc_distadmin.py:383 — DC was never being remapped to MD.
  • postprocessing/retail_rate_module/ferc_distadmin.py:449 — the diminishing-trend index column was never being clipped at zero.
  • postprocessing/retail_rate_module/retail_rate_calculations.py:879,885,922,926 — missing eval_period / depreciation_sch values were never being filled with the 20-year default, so they stayed NaN.
  • hourlize/reeds_to_rev.py:653 — values were not being filled with zero.

Deprecated kwargs (reeds/input_processing/recf.py:423,447) — dropped copy=False from pd.concat / .reindex. It had no effect but gave a Pandas4Warning.

Double-logged tracebacks (hourlize/reeds_to_rev.py:2166) — the handler logged every failure twice.

Known incompatibilities

These are pre-existing issues:

  • reeds_to_rev fails for every technology (81 of the remaining ERROR lines, plus 6 downstream [Errno 2] messages for missing df_sc_out_*_reduced.csv). It's probably worth turning off the reeds_to_rev switch until we get the script working again.
  • plot_max_imports fails with KeyError. firm_import_limit moved into inputs.h5 and uses transreg/allt, but reedsplots.py still expects *nercr/t. It is also plotted at level='nercr' while the limit is defined at transreg.

Validation, testing, and comparison report(s)

Full USA_defaults runs before and after:

before after
Total ERROR lines 554 81
retail_rate_calculations.py 469 0
recf.py 4 0
reeds_to_rev.py 81 81 (unchanged, out of scope)

By root cause:

Signature before after
ChainedAssignmentError 67 0
Pandas4Warning: copy keyword… 2 0
KeyError: …supply_curve_cost_per_mw 3 3
AttributeError: …no attribute 'FIPS' 1 1

After normalizing paths and case names, there are no substantive message differences between the "after" log the "before" log. retail_rate_calculations.py completed in both with an identical 19 INFO lines.

The only input-processing change is recf.py, and re-running it against an existing Pacific case produced byte-identical recf.h5, resources.csv, and csp.h5.

A compare report showing exactly no change for USA_Defaults is here: results-Main,LogUpdates.pptx.

Checklist for author

Details to double-check

  • Charge code provided to reviewers
  • Included comparison reports for appropriate test cases
  • [ ] Documentation updated if necessary
  • Code formatting standardized
  • Reusable functions used where possible instead of copy/pasted code

General information to guide review

  • Zero impact on results of default case (retail rate outputs are changed--other model results are not)
  • No large data file(s) added/modified
  • No substantive impact on runtime for full-US reference case
  • No substantive impact on folder size for full-US reference case
  • No change to process flow (runreeds.py, reeds/core/solve/solve.py)
  • No change to code organization
  • No change to package requirements (environment.yml or Project.toml)

Did you use LLM tools (chatbot or copilot) in the preparation of this PR? If so, describe how

Yes, I used Claude Opus 5 for this PR. It analyzed gamslog.txt against to identify each warning's origin, diagnosed the pandas 3 Copy-on-Write root cause, wrote the code changes, and ran verification checks using Pacific test cases. It also evaluated the before/after changes to USA_defaults gamslog.txt files. I also used it to produce the numbers and tables in this PR description.

Tag points of contact here if you would like additional review of the relevant parts of the model

@wesleyjcole
wesleyjcole merged commit 5203cbc into main Sep 14, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants