Repository navigation
[CU-1k5fpda7hax] Add omics workbench storage reindex - #194
Conversation
Re-scanning a connected storage bucket was only possible from the Reindex button in the Workbench Data tab, so node setup and clean-up (for example removing test samples from a customer node) could not be scripted from the omics CLI. The command starts the same background reindex; --wait polls the account until indexing is no longer RUNNING so a script can continue once the samples reflect the bucket. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011GSYqNdqNKNGKkpH2TRtPG
Workbench returns a failed indexing run to IDLE just like a finished one, so --wait would exit 0 and let a setup script carry on with samples that do not reflect the bucket. Workbench only advances last_indexed_at when a run completes, so --wait now reads it before triggering and fails unless it moved. An indexing status it does not recognise also fails instead of counting as finished, and a negative --timeout is rejected before any request is sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011GSYqNdqNKNGKkpH2TRtPG
The unknown-status and missing-status tests fail before the poll loop, so patching sleep there suggested they exercised it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011GSYqNdqNKNGKkpH2TRtPG
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified Files
|
🔍 Scrutiny — 8 findings: 2 serious, 6 moderateSecurity — security: 0 confirmed, 0 to validate, 0 refuted, 0 suppressed
Serious (2) 1. New help string exceeds repo's 120-char line length
Required by your REVIEW.md rule help='Wait for indexing to finish, then print the account. Exits non-zero if indexing did not complete.',CLAUDE.md's Linting > Configuration states the project uses a 120 character line length (configured in pyproject.toml). This new ArgumentSpec help string, introduced by the diff, is 121 characters long. Fix Shorten or wrap the help text so the line fits within 120 characters. Found by compliance · certain (100) · id compliance-1 2. Wait flow mixes escaped and unescaped id in one request chain
last_completed_at = client.get_storage_account(storage_id).last_indexed_at if wait else NoneBaseline GET and the poll loop send storage_id raw, while reindex_storage_account escapes it with path_segment; an id needing escaping breaks --wait before reindexing even starts. Fix Escape storage_id consistently across get, reindex, and poll calls. Found by correctness · certain (100) · rule no-partial-fixes · id correctness-1 Moderate (6) 3. New path_segment usage untested for actual escaping, unlike siblings
tests/unit/client/workbench/storage/test_storage_client.py:17-28 (test_posts_reindex_without_a_body_and_returns_the_running_account asserts only responses.calls[0].request.body is None, never the request path) vs tests/unit/client/workbench/ewes/test_ewes_engine_client.py:109-111 (test_engine_id_is_one_path_segment asserts session.delete.call_args.args[0] == f'{BASE}{NS}/engines/{ODD_SEGMENT}' for ODD_ID = 'a/../b#c?d') and tests/unit/client/workbench/workbench_user_service/test_workbench_user_client.py:115-127 (test_requests_user_as_one_path_segment parametrizes an id with special characters and asserts mock_session.get.assert_called_once_with(f'...{expected_path}'))The test only checks rejection of empty, '.', '..'; it never asserts the escaped request URL for an id with special characters. Fix Add a case with an id containing '/' and assert the percent-encoded request path. Found by parity · certain (100) · id parity-1 4. Indexing poll/failure-detection domain logic lives in the CLI entry adapter
# Workbench records last_indexed_at only when a run completes; a failed run returns to IDLE without it.
if account.last_indexed_at == last_completed_at:_wait_for_indexing encodes Workbench's indexing-state and completed-vs-failed semantics in commands.py instead of StorageClient, so no non-CLI caller can reuse it. Fix Move the poll/completion logic into StorageClient and raise a plain exception the CLI translates. Found by architecture · certain (100) · rule no-logic-in-entry-adapters · id architecture-2 5. No test covers indexing finishing before POST responds
while account.indexing_status == 'RUNNING':The while loop's zero-iteration success path, where POST already returns a finished IDLE account with a new last_indexed_at, is never exercised. Fix Add a test where POST already returns IDLE with a new last_indexed_at. Found by tests · certain (100) · rule new-behavior-test-coverage · id tests-1 6. Positive path-segment test never exercises real escaping
account = _make_client().reindex_storage_account('storage-1')The only passing reindex test posts id 'storage-1', which needs no percent-encoding, unlike the sibling user-client escaping test. Fix Add a case with a '/' or '%2E' id asserting the encoded path. Found by tests · certain (100) · rule new-behavior-test-coverage · id tests-2 7. No comment explains why only reindex escapes the storage id
urljoin(self.endpoint.url, f'{self.namespace}/storage/{path_segment(storage_account_id)}/reindex')reindex_storage_account calls path_segment on the id while get/update/delete interpolate it raw, with no note why. Fix Note why reindex alone needs path escaping, or apply it to the siblings. Found by documentation · certain (90) · rule explain-non-obvious-code · id documentation-3 8. Docstring overclaims account is always RUNNING after reindex
"""Start re-scanning a storage account's bucket. Indexing runs in the background; the returned account reports it as RUNNING until it finishes."""The docstring says the account always reports RUNNING, but commands.py treats a non-RUNNING account as already complete. Fix State the account may already be IDLE if indexing finished before the response. Found by documentation · likely (75) · id documentation-1 Scrutiny is an automated reviewer, and it never blocks a merge. Reply to any finding you think is wrong. Reviewed at commit 1d4108f; later commits are not reviewed. |
|
Review of 1d4108f, on top of the scrutiny report above: 0 blocking, merge-ready. The two scrutiny items marked serious are not blockers: the 121-char line passes The contract matches sample-service Not in the report, all optional:
|
reindex --wait reads the account before and while reindexing, so with only the reindex POST escaped an id that needs escaping failed on the first GET. Escaping it for get, update and delete too keeps one request chain consistent and stops a crafted id from addressing a path outside the account, as the engine client already does. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011GSYqHzB6MG7utV8vYMnZX
A Workbench that does not report indexing status could only be detected after the POST had already started indexing, and --timeout without --wait was silently ignored. Both now fail before any reindex is sent. The --wait help also says indexing keeps running when the wait ends early (timeout, poll error or Ctrl-C), and fits the 120-character line limit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011GSYqHzB6MG7utV8vYMnZX
…s in one test The reindex and get/update/delete rejection cases lived in two tests that each covered part of the method and id combinations; one parametrized test covers all of them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011GSYqHzB6MG7utV8vYMnZX
|
Thanks for the reviews. Pushed Escaping (security items, 2, 3, 6, 7, 8)
Not changed
Tests: 1376 passed, lint clean. |
Summary
Re-scanning a connected storage bucket is only possible from the Reindex button in the Workbench Data tab; operators need it from the
omicsCLI so they can script node setup and cleanup.omics workbench storage reindex <storage_id>calls sample-servicePOST /{namespace}/storage/{id}/reindex(requiresworkbench.storage.update, namespace admin) and prints the returned account.--waitpolls every 10 s and prints the account once indexing completes; it exits 1 on--timeout(default 3600 s; indexing keeps running), on an unrecognisedindexing_status, or on a failed run. A Workbench that does not reportindexing_status, or--timeoutwithout--wait, fails before any reindex is sent.StorageAccountgainsindexing_statusandlast_indexed_at.Decisions
last_indexed_atrather than the status, because sample-service returns a failed run toIDLEjust like a finished one and onlymarkIndexingCompletedsetslast_indexed_at. The command reads it with a GET before the POST, so a run that finishes before the POST responds still counts as completed.describe/list.path_segmentin every storage account request (get, update, delete, reindex), as the engine client does, so the--waitrequest chain is consistent.🤖 Generated with Claude Code
https://claude.ai/code/session_011GSYqNdqNKNGKkpH2TRtPG