refine: Implement --config by victorlin · Pull Request #2040 · nextstrain/augur

victorlin · 2026-08-15T01:27:41Z

Description of proposed changes

This PR implements augur refine --config, an option similar to augur subsample --config but for augur refine's own CLI options. See commits for details and #1987 for the motivation.

The implementation was written in a way that should make it easy to add --config to other commands in the future.

Docs preview: https://nextstrain--2040.org.readthedocs.build/projects/augur/en/2040/usage/cli/refine.html#configuration

Notable review threads

Drop no_covariance from config file? yes
Restrict some options (filepaths?) as CLI-only? add restrictions in workflow config schema

Checklist

Automated checks pass
Check if you need to add a changelog message
Check if you need to add tests
Check if you need to update docs
PR nextstrain.org for schema route

codecov · 2026-08-15T01:39:49Z

Codecov Report

❌ Patch coverage is 87.94326% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.54%. Comparing base (10c136e) to head (895e092).

Files with missing lines	Patch %	Lines
augur/argparse_.py	86.82%	8 Missing and 9 partials ⚠️

Additional details and impacted files

@@                          Coverage Diff                           @@
##           victorlin/resolve-input-file-paths    #2040      +/-   ##
======================================================================
+ Coverage                               74.39%   74.54%   +0.15%     
======================================================================
  Files                                      86       86              
  Lines                                   10743    10874     +131     
  Branches                                 2093     2123      +30     
======================================================================
+ Hits                                     7992     8106     +114     
- Misses                                   2365     2373       +8     
- Partials                                  386      395       +9

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

joverlee521

Kudos for designing a way to create the config schema from the parser! I left some open questions for discussion, but I do really like this direction.

jameshadfield

Really cool to see this Victor. I tried to dive as deep as I could, although argparse constantly confuses me! It was nice to see how simple this work made it to add this capability to further commands in #2041

jameshadfield · 2026-08-18T03:12:45Z

+        "metadata": {
+            "description": "sequence metadata",
+            "type": "string"
+        },


Aside: I imagine we'll use this schema for a general get_referenced_files function. That function (I think!) wants to see filepaths declared in the schema (if prop_schema.get("format") == "filepath") which are present in the subsampling schema:

augur/augur/data/schema-subsample-config.json

Lines 60 to 65 in 0aed098

"include": {

"oneOf": [

{

"type": "string",

"format": "filepath"

},

should we be adding them here?

Good point. There's nothing in the current argparse usage that marks arguments as filepaths, so I've added a custom InputFile type for the schema generating script:

augur/devel/regenerate-config-schemas

Lines 116 to 118 in 7838bb3

elif action.type is InputFile:

prop["type"] = "string"

prop["format"] = "filepath"

One related point -- some arguments are files or non-files. At least, I think we have some of them! We can leave their handling until we encounter them tho.

I've encountered this in #2041 for augur translate --genes. Leaving it as-is for now, but I think it would be clarifying to split into --genes/--genes-file so that we can type the latter as InputFile.

Yeah, I'm on board with that separation -- it'll make our lives much easier!

joverlee521

This is a good base for all of the other config based commands and I think it's in a good place to merge.

victorlin · 2026-08-31T23:21:57Z

Thanks for reviewing @joverlee521! As mentioned in dev chat, I think it'd be good to implement --config for commands listed in #1987 all at once. I'll update #2041 with commits for the other commands before merging here.

Enables configuration via a YAML file in addition to CLI arguments. All options available in the CLI are also available in the YAML file, but an option cannot be set in both simultaneously. Updated help text to reference both CLI and YAML syntax, now that both are valid options. Implemented using a new dependency, ConfigArgParse. It handles the YAML configuration within argparse so that the rest of the code can continue referencing the values from argparse. Lots of behavioral customization was added in CustomArgumentParser. Note that it doesn't make sense to use ConfigArgParse with augur subsample because the YAML config used in that command has a nested structure, opposed to the flat structure implicitly supported by the CLI. This commit just adds the functionality and a test. More to come in the following commits.

This will be useful as a basis for documentation, and validation by pathogen workflows. Done with a new script that inspects the argparse parser object so that all the logic and help text can stay in register_parser(). Added a new type=InputFile for argparse so that the schema can mark certain properties as filepaths.

The schema by itself isn't very useful for users. A table describing the available options is much more decipherable. Luckily we already do something for augur subsample's config docs. I've generalized an existing helper function to work with any flat JSON schema.

`covariance: False` accomplishes the same thing in a more intuitive syntax for the config file. Done by marking the option as CLI-only using a custom param in argparse's add_argument().

This was likely always the intention, but they were allowed together without the mutually exclusive group. The injection of cli_only in CustomArgumentParser.add_argument() is no longer sufficient as previously noted. Instead of monkeypatching the underlying add_argument() call, I chose to apply the same treatment to individual custom classes – more code but less hacky.

victorlin self-assigned this Aug 15, 2026

victorlin mentioned this pull request Aug 15, 2026

Config based Augur commands #1987

Open

7 tasks

victorlin force-pushed the victorlin/refine-config branch 2 times, most recently from 0d1ed80 to 8d72633 Compare August 17, 2026 17:52

joverlee521 reviewed Aug 18, 2026

View reviewed changes

Comment thread augur/data/schema-refine-config.json Outdated

Comment thread augur/data/schema-refine-config.json Outdated

Comment thread devel/regenerate-config-schemas

Comment thread augur/argparse_.py Outdated

victorlin mentioned this pull request Aug 18, 2026

Flexible rule configuration nextstrain/measles#146

Draft

2 tasks

victorlin force-pushed the victorlin/refine-config branch from 8d72633 to c0e909f Compare August 18, 2026 00:24

jameshadfield self-requested a review August 18, 2026 00:46

jameshadfield reviewed Aug 18, 2026

View reviewed changes

victorlin force-pushed the victorlin/refine-config branch 4 times, most recently from 3bdd7f3 to 11a6933 Compare August 20, 2026 00:36

victorlin force-pushed the victorlin/refine-config branch from 11a6933 to b3796ea Compare August 28, 2026 23:19

joverlee521 approved these changes Aug 31, 2026

View reviewed changes

Comment thread setup.py

victorlin force-pushed the victorlin/refine-config branch from b3796ea to 7531eae Compare September 1, 2026 23:27

victorlin mentioned this pull request Sep 1, 2026

Implement --config for various commands #2041

Open

3 of 5 tasks

victorlin force-pushed the victorlin/refine-config branch from 7531eae to 03c4e8d Compare September 2, 2026 00:05

victorlin mentioned this pull request Sep 3, 2026

Use ruamel.yaml #2046

Merged

4 tasks done

victorlin force-pushed the victorlin/refine-config branch 3 times, most recently from 09fe0a7 to aa9d0e7 Compare September 5, 2026 00:46

victorlin force-pushed the victorlin/refine-config branch 2 times, most recently from 5278ff5 to d56cf2f Compare September 18, 2026 22:11

victorlin force-pushed the victorlin/refine-config branch from d56cf2f to f0ebe4b Compare September 23, 2026 21:53

victorlin removed this pull request from stack #2042 September 23, 2026 22:01

victorlin changed the base branch from master to victorlin/resolve-input-file-paths September 23, 2026 22:01

victorlin added this pull request to stack #2052 September 23, 2026 22:01

victorlin force-pushed the victorlin/refine-config branch 2 times, most recently from 895e092 to b1c0439 Compare September 26, 2026 00:46

victorlin removed this pull request from stack #2052 September 26, 2026 00:46

victorlin changed the base branch from victorlin/resolve-input-file-paths to victorlin/remove-subsample-options September 26, 2026 00:47

victorlin added this pull request to stack #2054 September 26, 2026 00:47

victorlin added 7 commits September 25, 2026 17:50

Improve formatting of help message

9c6cadc

Add docs for --config

762a308

The schema by itself isn't very useful for users. A table describing the available options is much more decipherable. Luckily we already do something for augur subsample's config docs. I've generalized an existing helper function to work with any flat JSON schema.

Update changelog

c362c59

Remove no_covariance option from config file

62c4350

`covariance: False` accomplishes the same thing in a more intuitive syntax for the config file. Done by marking the option as CLI-only using a custom param in argparse's add_argument().

victorlin force-pushed the victorlin/refine-config branch from b1c0439 to 9c8f2f7 Compare September 26, 2026 00:51

	"include": {
	"oneOf": [
	{
	"type": "string",
	"format": "filepath"
	},

	elif action.type is InputFile:
	prop["type"] = "string"
	prop["format"] = "filepath"

Conversation

victorlin commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of proposed changes

Notable review threads

Checklist

Uh oh!

codecov Bot commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

joverlee521 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jameshadfield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jameshadfield Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

victorlin Aug 18, 2026

Choose a reason for hiding this comment

Uh oh!

jameshadfield Aug 19, 2026

Choose a reason for hiding this comment

Uh oh!

victorlin Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

jameshadfield Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

joverlee521 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

victorlin commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

victorlin commented Aug 15, 2026 •

edited

Loading

codecov Bot commented Aug 15, 2026 •

edited

Loading