feat(inventory): add resumable manifests

docs/commands.md

@@ -9,6 +9,8 @@ The commands available to you are:
 - `clear`: This removes all files belonging to the 'scope' and 'dataset', only
   available for the local and canfar sites.
 - `config`: Edit the `.datatrail/config.yaml` configuration file.
+- `inventory`: Recursively discover datasets and write their file replica URIs
+  to a resumable JSON manifest.
 - `list`: This list either the 'scopes' available or all of the datasets
   belonging to the given dataset.
 - `ps`: This provides detailed information for the given 'scope' and 'dataset' combination.

docs/inventory.md

@@ -0,0 +1,71 @@
+# Building a durable inventory
+
+`datatrail inventory` recursively discovers terminal datasets and records each
+file replica URI in a versioned JSON manifest. The manifest can later drive a
+batch download without repeating the archive crawl.
+
+Every run must be bounded in one of these ways:
+
+```shell
+$> datatrail inventory chime.event.baseband.raw
+$> datatrail inventory --match classified,baseband
+$> datatrail inventory gbo.acquisition.processed --parent complex_gains
+```
+
+A scope limits traversal to that scope. `--match` may search across scopes but
+only opens matching larger datasets. `--parent` requires a scope and starts at
+one known dataset. It cannot be combined with `--match`.
+
+Use `--output` to choose the manifest path:
+
+```shell
+$> datatrail inventory chime.event.baseband.raw --match classified \
+     --output baseband-inventory.json
+```
+
+The command writes the manifest atomically after discovery and after every
+dataset query. A rerun with the same selection reuses `ready` and `empty`
+entries, then retries `pending` and `failed` entries. A different selection is
+refused so unrelated inventories cannot be mixed.
+
+An inventory with any failed discovery branch or file query exits nonzero.
+`--allow-incomplete` keeps the incomplete manifest but exits zero when a caller
+wants to inspect or process the available subset.
+
+## Manifest format
+
+The first format is `datatrail.inventory/v1`:
+
+```json
+{
+  "schema": "datatrail.inventory/v1",
+  "selection": {
+    "scope": "gbo.acquisition.processed",
+    "match": [],
+    "parent": "complex_gains"
+  },
+  "complete": true,
+  "discovery_failures": [],
+  "datasets": [
+    {
+      "scope": "gbo.acquisition.processed",
+      "dataset": "20230525",
+      "parent": "complex_gains",
+      "path": ["complex_gains", "20230525"],
+      "status": "ready",
+      "replicas": [
+        {
+          "storage_element": "minoc",
+          "uri": "cadc:CHIMEFRB/example/file.h5"
+        }
+      ]
+    }
+  ]
+}
+```
+
+Replica rows contain only Datatrail information. Size and checksum fields may
+be added in a later schema when they can be obtained without requiring a CADC
+credential during inventory creation. A valid dataset with no replica URIs has
+status `empty`; an unavailable or invalid file response has status `failed`
+and an `error` field.

docs/list.md

@@ -12,6 +12,12 @@ Options:
   -q, --quiet    Only errors shown in logs.
   --write        Write the events to file.
   --json         Output as JSON.
+  --match TEXT   Comma-separated, case-insensitive terms a larger dataset must
+                 all contain.
+  --expand       Open each matched larger dataset one level and list its
+                 children.
+  --recursive    Open each matched larger dataset through all descendant
+                 levels.
   --help         Show this message and exit.
 
 ```
@@ -102,6 +108,104 @@ Within Datatrail, there are two types of datasets:
     Please see the CLI reference page for more information on the `list` command:
     [datatrail list](../cli/#datatrail-list)
 
+## Finding datasets with `--match` and `--expand`
+
+Navigating the hierarchy one name at a time gets slow when you do not know
+where a dataset lives. `--match` filters the larger datasets of a scope, or
+of **every** scope when no scope is given, by one or more comma-separated,
+case-insensitive terms, which must all appear in the combined
+`scope dataset` text:
+
+```shell
+$> datatrail ls chime.acquisition.processed --match gains
+      Datatrail: Dataset Map
++-----------------------------+---------------+
+| Scope                       | Dataset       |
++-----------------------------+---------------+
+| chime.acquisition.processed | complex_gains |
++-----------------------------+---------------+
+```
+
+A hit may be a container whose children are the datasets you actually want.
+`--expand` opens each matched larger dataset one level and lists the children
+it finds, recording the parent; a match whose children cannot be listed keeps
+its own row:
+
+```shell
+$> datatrail ls --match gain --expand
+                    Datatrail: Dataset Map
++-----------------------------+---------------+---------------+
+| Scope                       | Dataset       | Parent        |
++-----------------------------+---------------+---------------+
+| chime.acquisition.processed | complex_gains |               |
++-----------------------------+---------------+---------------+
+| gbo.acquisition.processed   | 20230716      | complex_gains |
+| gbo.acquisition.processed   | 20230715      | complex_gains |
+| ...                         | ...           | ...           |
++-----------------------------+---------------+---------------+
+```
+
+Rows reached through a parent resolve directly with
+`datatrail ps <scope> <dataset>`; a row kept for a matched dataset whose
+children could not be listed (or that has none) may still be a container.
+
+!!! warning "Incomplete maps"
+
+    If Datatrail does not answer for a scope or dataset during the walk, the
+    map is reported as **incomplete** and the unanswered queries are listed,
+    rather than silently showing them as empty. With `--json`, those queries
+    appear in the `failed` list. A partial map still exits 0; a map with **no**
+    rows and unanswered queries exits 1, since nothing was determined.
+
+`--recursive` follows every descendant of each matched larger dataset instead
+of stopping after one level. It emits terminal datasets and records the full
+path used to reach each one:
+
+```shell
+$> datatrail ls gbo.acquisition.processed --match gains --recursive
+```
+
+The walk visits each dataset once, so shared descendants are not duplicated
+and hierarchy cycles cannot loop forever. The first path found in sorted order
+is retained. An answered empty child list is a terminal dataset. A branch that
+does not answer is retained as a partial row and also listed under `failed`.
+Like `--expand`, a recursive walk across all scopes requires `--match` to keep
+the request bounded.
+
+With `--json`, the map is emitted as structured rows for scripting; `parent`
+is `null` for rows that were not reached through expansion. Recursive rows
+also include `path`, from the matched larger dataset through the terminal row:
+
+```bash
+$ datatrail ls --match gain --expand --json
+{
+  "results": [
+    {
+      "scope": "gbo.acquisition.processed",
+      "dataset": "20230525",
+      "parent": "complex_gains"
+    },
+    ...
+  ],
+  "failed": []
+}
+```
+
+```bash
+$ datatrail ls gbo.acquisition.processed --match gains --recursive --json
+{
+  "results": [
+    {
+      "scope": "gbo.acquisition.processed",
+      "dataset": "20230525",
+      "parent": "complex_gains",
+      "path": ["complex_gains", "20230525"]
+    }
+  ],
+  "failed": []
+}
+```
+
 ## 🤖 Machine-readable JSON output
 
 The `--json` flag outputs structured JSON instead of formatted tables, making it easy to parse the output in scripts and pipelines:

dtcli/cli.py

@@ -6,7 +6,7 @@
 from click_aliasing import ClickAliasedGroup
 from rich import console, pretty
 
-from dtcli import clear, config, ls, ps, pull, scout, unregistered
+from dtcli import clear, config, inventory, ls, ps, pull, scout, unregistered
 from dtcli.utilities import utilities
 
 pretty.install()
@@ -43,6 +43,7 @@ def version():
 
 cli.add_command(clear.clear)
 cli.add_command(config.config)
+cli.add_command(inventory.inventory)
 cli.add_command(ls.list, aliases=["ls"])
 cli.add_command(ps.ps)
 cli.add_command(pull.pull)