Skip to content

[CU-1k5fpda7hax] Add omics workbench storage reindex - #194

Merged
Kevin-DNA merged 6 commits into
mainfrom
storage-reindex-1k5fpda7hax
Nov 16, 2026
Merged

Kevin-DNA merged 6 commits into
mainfrom
storage-reindex-1k5fpda7hax

Conversation

@Kevin-DNA

@Kevin-DNA Kevin-DNA commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Re-scanning a connected storage bucket is only possible from the Reindex button in the Workbench Data tab; operators need it from the omics CLI so they can script node setup and cleanup. omics workbench storage reindex <storage_id> calls sample-service POST /{namespace}/storage/{id}/reindex (requires workbench.storage.update, namespace admin) and prints the returned account. --wait polls every 10 s and prints the account once indexing completes; it exits 1 on --timeout (default 3600 s; indexing keeps running), on an unrecognised indexing_status, or on a failed run. A Workbench that does not report indexing_status, or --timeout without --wait, fails before any reindex is sent. StorageAccount gains indexing_status and last_indexed_at.

Decisions

  • Detect a failed run from last_indexed_at rather than the status, because sample-service returns a failed run to IDLE just like a finished one and only markIndexingCompleted sets last_indexed_at. The command reads it with a GET before the POST, so a run that finishes before the POST responds still counts as completed.
  • Type the new fields as plain strings rather than an enum, so a new server status does not break describe/list.
  • Escape the id with path_segment in every storage account request (get, update, delete, reindex), as the engine client does, so the --wait request chain is consistent.

🤖 Generated with Claude Code

https://claude.ai/code/session_011GSYqNdqNKNGKkpH2TRtPG

Kevin-DNA and others added 3 commits November 16, 2026 10:05
Re-scanning a connected storage bucket was only possible from the Reindex
button in the Workbench Data tab, so node setup and clean-up (for example
removing test samples from a customer node) could not be scripted from the
omics CLI. The command starts the same background reindex; --wait polls
the account until indexing is no longer RUNNING so a script can continue
once the samples reflect the bucket.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011GSYqNdqNKNGKkpH2TRtPG
Workbench returns a failed indexing run to IDLE just like a finished one,
so --wait would exit 0 and let a setup script carry on with samples that
do not reflect the bucket. Workbench only advances last_indexed_at when a
run completes, so --wait now reads it before triggering and fails unless
it moved. An indexing status it does not recognise also fails instead of
counting as finished, and a negative --timeout is rejected before any
request is sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011GSYqNdqNKNGKkpH2TRtPG
The unknown-status and missing-status tests fail before the poll loop, so
patching sleep there suggested they exercised it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011GSYqNdqNKNGKkpH2TRtPG
@platform-automation-dnastack

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines Covered Coverage Threshold Status
12330 8039 65% 30% 🟢

New Files

No new covered files...

Modified Files

File Coverage Status
dnastack/cli/commands/workbench/storage/commands.py 78% 🟢
dnastack/client/workbench/storage/client.py 87% 🟢
dnastack/client/workbench/storage/models.py 98% 🟢
TOTAL 88% 🟢

updated for commit: 9d7315a by action🐍

@Kevin-DNA
Kevin-DNA marked this pull request as ready for review November 16, 2026 02:13
@mpanik-dnastack

Copy link
Copy Markdown

🔍 Scrutiny — 8 findings: 2 serious, 6 moderate

Security — security: 0 confirmed, 0 to validate, 0 refuted, 0 suppressed

  • pre-existing dnastack/client/workbench/storage/client.py:80 Unescaped storage_id lets a crafted id traverse out of the namespace [id adversary-insider-1]
  • pre-existing dnastack/client/workbench/storage/client.py:80 New --wait path reuses unescaped get_storage_account URL sink [id adversary-outsider-1]
  • pre-existing dnastack/client/workbench/storage/client.py:80 get_storage_account splices storage_id unescaped, enabling path traversal [id security-data-isolation-1]

Serious (2)

1. New help string exceeds repo's 120-char line length

dnastack/cli/commands/workbench/storage/commands.py:168

Required by your REVIEW.md rule CLAUDE.md Linting > Configuration: "Minimal setup in pyproject.toml with Python 3.11 target and 120 character line length".

help='Wait for indexing to finish, then print the account. Exits non-zero if indexing did not complete.',

CLAUDE.md's Linting > Configuration states the project uses a 120 character line length (configured in pyproject.toml). This new ArgumentSpec help string, introduced by the diff, is 121 characters long.

Fix Shorten or wrap the help text so the line fits within 120 characters.

Found by compliance · certain (100) · id compliance-1

2. Wait flow mixes escaped and unescaped id in one request chain

dnastack/cli/commands/workbench/storage/commands.py:199

last_completed_at = client.get_storage_account(storage_id).last_indexed_at if wait else None

Baseline GET and the poll loop send storage_id raw, while reindex_storage_account escapes it with path_segment; an id needing escaping breaks --wait before reindexing even starts.

Fix Escape storage_id consistently across get, reindex, and poll calls.

Found by correctness · certain (100) · rule no-partial-fixes · id correctness-1

Moderate (6)

3. New path_segment usage untested for actual escaping, unlike siblings

tests/unit/client/workbench/storage/test_storage_client.py:14-28

tests/unit/client/workbench/storage/test_storage_client.py:17-28 (test_posts_reindex_without_a_body_and_returns_the_running_account asserts only responses.calls[0].request.body is None, never the request path) vs tests/unit/client/workbench/ewes/test_ewes_engine_client.py:109-111 (test_engine_id_is_one_path_segment asserts session.delete.call_args.args[0] == f'{BASE}{NS}/engines/{ODD_SEGMENT}' for ODD_ID = 'a/../b#c?d') and tests/unit/client/workbench/workbench_user_service/test_workbench_user_client.py:115-127 (test_requests_user_as_one_path_segment parametrizes an id with special characters and asserts mock_session.get.assert_called_once_with(f'...{expected_path}'))

The test only checks rejection of empty, '.', '..'; it never asserts the escaped request URL for an id with special characters.

Fix Add a case with an id containing '/' and assert the percent-encoded request path.

Found by parity · certain (100) · id parity-1

4. Indexing poll/failure-detection domain logic lives in the CLI entry adapter

dnastack/cli/commands/workbench/storage/commands.py:19-44

# Workbench records last_indexed_at only when a run completes; a failed run returns to IDLE without it.
    if account.last_indexed_at == last_completed_at:

_wait_for_indexing encodes Workbench's indexing-state and completed-vs-failed semantics in commands.py instead of StorageClient, so no non-CLI caller can reuse it.

Fix Move the poll/completion logic into StorageClient and raise a plain exception the CLI translates.

Found by architecture · certain (100) · rule no-logic-in-entry-adapters · id architecture-2

5. No test covers indexing finishing before POST responds

dnastack/cli/commands/workbench/storage/commands.py:29

while account.indexing_status == 'RUNNING':

The while loop's zero-iteration success path, where POST already returns a finished IDLE account with a new last_indexed_at, is never exercised.

Fix Add a test where POST already returns IDLE with a new last_indexed_at.

Found by tests · certain (100) · rule new-behavior-test-coverage · id tests-1

6. Positive path-segment test never exercises real escaping

tests/unit/client/workbench/storage/test_storage_client.py:23

account = _make_client().reindex_storage_account('storage-1')

The only passing reindex test posts id 'storage-1', which needs no percent-encoding, unlike the sibling user-client escaping test.

Fix Add a case with a '/' or '%2E' id asserting the encoded path.

Found by tests · certain (100) · rule new-behavior-test-coverage · id tests-2

7. No comment explains why only reindex escapes the storage id

dnastack/client/workbench/storage/client.py:88

urljoin(self.endpoint.url, f'{self.namespace}/storage/{path_segment(storage_account_id)}/reindex')

reindex_storage_account calls path_segment on the id while get/update/delete interpolate it raw, with no note why.

Fix Note why reindex alone needs path escaping, or apply it to the siblings.

Found by documentation · certain (90) · rule explain-non-obvious-code · id documentation-3

8. Docstring overclaims account is always RUNNING after reindex

dnastack/client/workbench/storage/client.py:84-85

"""Start re-scanning a storage account's bucket. Indexing runs in the background; the returned account reports it as RUNNING until it finishes."""

The docstring says the account always reports RUNNING, but commands.py treats a non-RUNNING account as already complete.

Fix State the account may already be IDLE if indexing finished before the response.

Found by documentation · likely (75) · id documentation-1

Scrutiny is an automated reviewer, and it never blocks a merge. Reply to any finding you think is wrong. Reviewed at commit 1d4108f; later commits are not reviewed.

@mpanik-dnastack

Copy link
Copy Markdown

Review of 1d4108f, on top of the scrutiny report above: 0 blocking, merge-ready. The two scrutiny items marked serious are not blockers: the 121-char line passes make lint (ruff does not select E501 here), and the mixed escaping fails on the first GET, before any reindex starts.

The contract matches sample-service main: POST /{namespace}/storage/{id}/reindex needs workbench.storage.update, returns the account with indexing_status (IDLE/RUNNING) and nullable last_indexed_at, and only markIndexingCompleted sets last_indexed_at, so a failed run returns to IDLE without it, as the PR says. Unit suite: 1361 passed, 2 skipped.

Not in the report, all optional:

  • Help text: reindexing deletes sample files whose bucket objects are gone (AbstractPlatformIndexer.deleteOrphanedSampleFiles). One sentence in the help would warn people scripting it.
  • --wait rough edges (seen against a local fake server, not a live one):
    • a single 502 while polling ends --wait with a raw ServerError, no retry;
    • neither that nor Ctrl-C says indexing keeps running server-side;
    • --timeout without --wait is silently ignored;
    • if the server omits indexing_status, --wait exits 1 only after the POST was sent.

Kevin-DNA and others added 3 commits November 16, 2026 21:26
reindex --wait reads the account before and while reindexing, so with only
the reindex POST escaped an id that needs escaping failed on the first GET.
Escaping it for get, update and delete too keeps one request chain
consistent and stops a crafted id from addressing a path outside the
account, as the engine client already does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011GSYqHzB6MG7utV8vYMnZX
A Workbench that does not report indexing status could only be detected
after the POST had already started indexing, and --timeout without --wait
was silently ignored. Both now fail before any reindex is sent. The --wait
help also says indexing keeps running when the wait ends early (timeout,
poll error or Ctrl-C), and fits the 120-character line limit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011GSYqHzB6MG7utV8vYMnZX
…s in one test

The reindex and get/update/delete rejection cases lived in two tests that
each covered part of the method and id combinations; one parametrized test
covers all of them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011GSYqHzB6MG7utV8vYMnZX
@Kevin-DNA

Copy link
Copy Markdown
Contributor Author

Thanks for the reviews. Pushed 7c217c4, 1ca71da, 9d7315a. The head is now 9d7315a.

Escaping (security items, 2, 3, 6, 7, 8)

  • All per-account calls (get, update, delete, reindex) now build their URL through StorageClient._account_url, which escapes the ID with path_segment, the same as EwesClient._engine_url. '', . and .. are rejected before any request. For normal IDs the request URLs are unchanged.
  • A new test sends a/../b#c?d through all four and asserts it stays one segment.
  • The docstring now says the account reports RUNNING, or IDLE if indexing already finished before the response.

--wait / --timeout (@mpanik-dnastack, items 1, 5)

  • --timeout without --wait now fails with exit 2 before any request.
  • A missing indexing_status is caught on the first GET, before the reindex is sent.
  • The help says indexing keeps running if the wait ends early, and it now fits 120 characters.
  • Added a test for indexing that finishes before the POST returns.

Not changed

  • Moving the polling into the client (4): nothing outside the CLI calls it yet. It's a mechanical move once there's a second caller.
  • Retry on a 502 while polling: retries belong in the shared HTTP session rather than one command. The help now makes clear that a failed wait doesn't mean a failed reindex.
  • Warning that reindex removes orphaned sample files: it's already in the command help ("Indexing also removes sample files whose objects are no longer in the bucket").

Tests: 1376 passed, lint clean.

@Kevin-DNA
Kevin-DNA merged commit 894b471 into main Nov 16, 2026
6 checks passed
@Kevin-DNA
Kevin-DNA deleted the storage-reindex-1k5fpda7hax branch November 16, 2026 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants