Skip to content

feat(deposition): split and move out functions to be reused in raw reads, improve manifest creation code, fix bug in assembly revision code - #7017

Open
anna-parker wants to merge 8 commits into
mainfrom
small_ena_refactor
Open

feat(deposition): split and move out functions to be reused in raw reads, improve manifest creation code, fix bug in assembly revision code#7017
anna-parker wants to merge 8 commits into
mainfrom
small_ena_refactor

Conversation

@anna-parker

@anna-parker anna-parker commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

refactor in preparation of #6816

  • splits and moves some functions out of the assembly file into generic helper files so that functions can be reused in raw reads
  • improves manifest creation code (raw reads also require a manifest with slightly different fields)
  • fix bug in assembly revision code: if allow_revision_with_manifest_changes is true we also want the pipeline to revise assemblies automatically (instead of sending us a slack notification) if the manifest metadata changed. I added this to the ena workflow tests.

Screenshot

PR Checklist

  • All necessary documentation has been adapted.
  • The implemented feature is covered by appropriate, automated tests.
    I added additional unit tests for the manifest generation and manifest field diff code and additionally I added a new ena workflow test that covers the bug case (workflow tests talk to ENA dev directly, sadly ENA dev does not process assemblies so part of the full process can only be covered in production code but the workflow tests document this)

🚀 Preview: Add preview label to enable

@claude claude Bot added the deposition related to ENA/INSDC deposition label Jul 31, 2026
@claude

claude Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

This PR may be related to: #4347 (Raw read epic), #4353 (MVP raw reads automate ENA deposition), #6877 (Handle raw reads revision on ENA) — this refactor is explicit prep work for #6816, which addresses these.

return sorted(segment_order)


def get_address(config: Config, submission_row: SubmissionTableEntry) -> str | None:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved

return address_string


def get_assembly_values_in_metadata(config: Config, metadata: dict[str, str]) -> dict[str, str]:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replaced with resolve_required_manifest_field

f"for {submission_row.accession}. (Maybe other fields changed as well)"
)
return True
if config.allow_revision_with_manifest_changes and manifest_fields_diff(

@anna-parker anna-parker Jul 31, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was actually a bug - if allow_revision_with_manifest_changes is true we also want to revise assemblies if the manifest metadata changed

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a test for this!

)


def get_project_and_sample_results(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved

@anna-parker
anna-parker marked this pull request as ready for review July 31, 2026 20:23
@claude

claude Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Claude finished @anna-parker's task in 4m 32s —— View job


Claude finished reviewing this PR

  • Gather context (PR diff, changed files)
  • Read changed files in full
  • Review code quality, security, performance, testing, docs
  • Post inline comments for actionable issues
  • Post final summary review

Summary

This is a clean refactor: extracting get_address, get_last_entry, get_project_and_sample_results, manifest_fields_diff, linked_accession_diff, resolve_manifest_field/resolve_required_manifest_field out of create_assembly.py so they can be reused by the upcoming raw-reads deposition work (#6816). The generic field-writing loop in create_manifest (iterating dataclasses.fields(manifest)) is a nice simplification and is safe because the manifest file is an unordered key-value TSV (confirmed by the existing test_create_manifest test, which parses into a dict rather than asserting line order).

Also good fixes bundled in here:

  • is_flatfile_data_changed (create_assembly.py:407) now also checks manifest_fields_diff when allow_revision_with_manifest_changes=True, closing a gap where manifest-only changes could be silently skipped on revision — flagged by the author as an actual bug fix.
  • notifications.py: notify() now guards requests.post with try/except so a Slack outage doesn't crash the pipeline.
  • Moving the fasta/flatfile presence check in create_manifest above directory creation avoids leaving an empty manifest dir behind on error.

I left a few inline comments on smaller items:

  1. resolve_manifest_field (ena_submission_helper.py:563) — the new empty-string handling for multi-field mappings differs subtly from the old "filter falsy values" behavior; could produce a trailing ", " instead of falling back to default.
  2. resolve_manifest_field (ena_submission_helper.py:561) — reformat_authors only reformats values[0], silently dropping any additional configured loculus_fields (not an issue with the current single-field authors mapping, but worth a guard if this generalizes).
  3. Testing gap (ena_submission_helper.py:581) — the newly extracted functions (resolve_manifest_field, resolve_required_manifest_field, manifest_fields_diff, linked_accession_diff) have no direct unit tests; only create_manifest_object/create_manifest are exercised end-to-end.
  4. coverage field default/type change (config/defaults.yaml:43) — dropping type: int changes the formatting of the COVERAGE manifest value from a truncated integer to a raw string (could now contain decimals); likely fine/safer, but untested.

Nothing blocking — these are refinements, not correctness-breaking issues. ena-submission-unit-tests is already green in CI.
· Branch: small_ena_refactor

Comment thread ena-submission/src/ena_deposition/ena_submission_helper.py Outdated
Comment thread ena-submission/src/ena_deposition/ena_submission_helper.py
Comment thread ena-submission/src/ena_deposition/ena_submission_helper.py
Comment thread ena-submission/config/defaults.yaml
@anna-parker anna-parker changed the title feat(deposition): split and move out functions to be reused in raw reads, improve manifest creation code feat(deposition): split and move out functions to be reused in raw reads, improve manifest creation code, fix bug in assembly revision code Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deposition related to ENA/INSDC deposition

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant