Fix log file errors - #211
Merged
Merged
Conversation
kennedy-mindermann
approved these changes
Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A typical
gamslog.txtfile has manyERRORentries (my USA_default case from main had 554 entries). These have sometimes confused users, especially because these runs have completed successfully. This pull request cleans up most of theERRORand warning noise ingamslog.txt, and fixes the pandas 3 Copy-on-Write (CoW) bugs that were associated with some of those messages. The remaining issues are primarily fromreeds_to_rev.pywhich is currently not funtional.Technical details
Implementation notes
Logging (
reeds/log.py) —makelog()redirectssys.stderrinto the logger atlogging.ERROR. Python'swarningsmodule writes to stderr, so every warning was labeledERROR. Addinglogging.captureWarnings(True)sendswarnings.warn()through the logger atWARNINGinstead.Chained assignment (retail rate module,
reeds_to_rev.py) — Indf['col'].fillna(..., inplace=True),df['col']returns a temporary Series and the mutation is applied to that temporary Series, which is then discarded. Pre-CoW,df['col']was often a "view", so the write happened to propagate. pandas 3 makes CoW unconditional, so it no longer propogates. Specific issues:postprocessing/retail_rate_module/ferc_distadmin.py:383— DC was never being remapped to MD.postprocessing/retail_rate_module/ferc_distadmin.py:449— the diminishing-trendindexcolumn was never being clipped at zero.postprocessing/retail_rate_module/retail_rate_calculations.py:879,885,922,926— missingeval_period/depreciation_schvalues were never being filled with the 20-year default, so they stayed NaN.hourlize/reeds_to_rev.py:653— values were not being filled with zero.Deprecated kwargs (
reeds/input_processing/recf.py:423,447) — droppedcopy=Falsefrompd.concat/.reindex. It had no effect but gave aPandas4Warning.Double-logged tracebacks (
hourlize/reeds_to_rev.py:2166) — the handler logged every failure twice.Known incompatibilities
These are pre-existing issues:
reeds_to_revfails for every technology (81 of the remainingERRORlines, plus 6 downstream[Errno 2]messages for missingdf_sc_out_*_reduced.csv). It's probably worth turning off thereeds_to_revswitch until we get the script working again.plot_max_importsfails withKeyError.firm_import_limitmoved intoinputs.h5and usestransreg/allt, butreedsplots.pystill expects*nercr/t. It is also plotted atlevel='nercr'while the limit is defined attransreg.Validation, testing, and comparison report(s)
Full
USA_defaultsruns before and after:ERRORlinesretail_rate_calculations.pyrecf.pyreeds_to_rev.pyBy root cause:
ChainedAssignmentErrorPandas4Warning: copy keyword…KeyError: …supply_curve_cost_per_mwAttributeError: …no attribute 'FIPS'After normalizing paths and case names, there are no substantive message differences between the "after" log the "before" log.
retail_rate_calculations.pycompleted in both with an identical 19INFOlines.The only input-processing change is
recf.py, and re-running it against an existing Pacific case produced byte-identicalrecf.h5,resources.csv, andcsp.h5.A compare report showing exactly no change for USA_Defaults is here: results-Main,LogUpdates.pptx.
Checklist for author
Details to double-check
[ ] Documentation updated if necessaryGeneral information to guide review
Did you use LLM tools (chatbot or copilot) in the preparation of this PR? If so, describe how
Yes, I used Claude Opus 5 for this PR. It analyzed
gamslog.txtagainst to identify each warning's origin, diagnosed the pandas 3 Copy-on-Write root cause, wrote the code changes, and ran verification checks using Pacific test cases. It also evaluated the before/after changes to USA_defaultsgamslog.txtfiles. I also used it to produce the numbers and tables in this PR description.Tag points of contact here if you would like additional review of the relevant parts of the model