diff --git a/CHANGELOG.md b/CHANGELOG.md
index 81fa09a..a6c1dab 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,46 @@ All notable changes to this project will be documented in this file
## 2.8.0 (unreleased)
+- New `rspace_client.fs`: RSpace as an [fsspec](https://filesystem-spec.readthedocs.io)
+ filesystem. `RSpaceFilesystem` mounts `gallery/`, `inventory/` and `workspace/` (Gallery
+ folders and files; Inventory benches, containers, samples, subsamples, templates and
+ instruments as folders of their records and attachments, attachment fields as one-file
+ folders; ELN folders, notebooks and documents, a document as a folder of its text fields
+ and a field as a folder of its linked files). Paths are record names; duplicate names
+ and names that read as addresses carry their global ID (`data [GL112].csv`);
+ `path_style="labelled"` / `"id"` for stable paths. Read-only by default (`writable=True`,
+ `allow_delete=True`). Uploads into Inventory and the Workspace go through the Gallery and
+ link; `cp_file` from any Gallery-backed file links instead of copying bytes. Transactions
+ defer uploads. `rspace://host` URLs via `fsspec.open`, `fsspec.filesystem("rspace")` and
+ `UPath` (registered as an `fsspec.specs` entry point), key from `RSPACE_API_KEY` only. `glob("
/**")` answers Galaxy's search
+ including bracketed names. New dependency `fsspec`. Listings recognise the Gallery root by its `GF`
+ id, and refuse a page without its records or a `next` link that loops. A search typed as `*.csv` or
+ `data*` works through Galaxy's search box instead of failing on the `**` it produces.
+ Linking or copying into an occupied Inventory attachment field is refused like an upload,
+ instead of replacing its file. Field edits in the Workspace take a per-document lock and are
+ read back and re-applied when another writer overwrote them, so concurrent uploads into one
+ field no longer drop links (writers in separate processes can still race; the API has no
+ atomic append). A file segment written as a global ID only resolves under the field, record or
+ Gallery folder that really holds it, and a missing one is `FileNotFoundError`.
+- An error answered with RSpace's HTML error page is reported by the page's title instead of
+ the whole page.
+- Deprecated `rspace_client.eln.fs.GalleryFilesystem` and
+ `rspace_client.inv.attachment_fs.InventoryAttachmentFilesystem` (PyFilesystem2, unmaintained
+ upstream). Unchanged in behaviour apart from the fixes below, now emit a `DeprecationWarning`; removal together with
+ the `fs` dependency in 3.0, after Galaxy's shipped plugin has moved to `rspace_client.fs`.
+- Fix: uploads accept a file object that forwards `read` through `__getattr__`. requests 2.34
+ refuses such objects on Python 3.12+, which broke exports from Galaxy 26.1 (it pins requests
+ 2.34.2) to RSpace through Galaxy's shipped `rspace` file source.
+- Fix: `GalleryFilesystem` reports a file's `created` date as `+00:00` instead of RSpace's
+ trailing `Z` (the same UTC instant). Galaxy's shipped `rspace` file source parses it with
+ `datetime.fromisoformat`, which rejects `Z` before Python 3.11, so on Python 3.10 Galaxy
+ failed to list any Gallery folder holding files.
+- `rspace_client/tests/galaxy_plugin_contract_test.py` pins what Galaxy's shipped plugin relies
+ on; a failure there is a release blocker.
+- Tests: the Link-field integration tests always run instead of needing
+ `RSPACE_SUPPORTS_LINK_FIELDS=1`; current RSpace releases support Link fields.
+- `tools/galaxy/rspace_fsspec_source.py`: the reference Galaxy file source for the new
+ filesystem, to be proposed upstream.
- Security: the client never follows an HTTP redirect, since `requests` would carry the
`apiKey` header to the new host.
- Client-level changes: `_links` URLs are rebased onto the configured base URL, path prefix
diff --git a/DEVELOPING.md b/DEVELOPING.md
index bef4d7a..cc26d84 100644
--- a/DEVELOPING.md
+++ b/DEVELOPING.md
@@ -58,6 +58,10 @@ All top-level methods for use by client code should be unit-tested.
RSpace can be run on Docker on a developer machine, providing access to a sandbox environment.
+#### Galaxy's RSpace file source
+
+Galaxy ships an `rspace` file source built on this library's deprecated `rspace_client.eln.fs.GalleryFilesystem`, so changes here can break Galaxy. `rspace_client/tests/galaxy_plugin_contract_test.py` pins what it relies on and runs with the normal suite; a failure there is a release blocker. [tools/galaxy/README.md](tools/galaxy/README.md) shows how to drive Galaxy's real plugin without a Galaxy server, and how to try the fsspec file source proposed for Galaxy. Why the contract looks the way it does is in [docs/fsspec-filesystem-design.md](docs/fsspec-filesystem-design.md), section 8.
+
### Making a release
diff --git a/README.md b/README.md
index 28a4076..4b7cf06 100644
--- a/README.md
+++ b/README.md
@@ -45,7 +45,7 @@ Full REST API reference is served by your own RSpace instance at `https:///**` shape: one listing of the directory, an
+ unescaped case-insensitive substring match on the last segment, results in listing order.
+
+Inventory adds fixed section names at its root, so each branch has a real resolver rather than
+the old "strip the two-letter prefix" helper. Nothing in a path is a page token: `page-N`
+pseudo-folders were built and removed, see section 5.
+
+## 3. Write posture
+
+Read-only by default. `writable=True` allows uploads, folder creation and links;
+`allow_delete=True` allows removals. The two are separate on purpose, so `read_only` is false
+if either is set. Every mutating method checks the posture before any request and raises
+`ReadOnlyError` (a `PermissionError`), and the fixed top level of `RSpaceFilesystem` refuses
+writes the same way, so a write there cannot silently succeed. A `move` checks the delete permission before copying anything.
+
+`remove` deliberately differs per branch: in the Workspace it unlinks the file from its field
+and leaves the Gallery file alone; in the Inventory it deletes the attachment record; in the
+Gallery it is unsupported, because the API cannot delete Gallery files. Each says which it did.
+
+A signed ELN document is locked. Every field write (upload, link, remove) refuses before any
+request is made, and a document's fields are listed with a `(signed)` suffix under the `name`
+style and report the lock as `writable: false` in their entry. The folder-tree endpoint does not carry
+the flag, so the document's own entry in its parent listing is not marked.
+
+Uploading into a text field is a read-modify-write of the field's HTML; concurrent edits are
+last-writer-wins, as they are for the web client.
+
+## 4. What each branch contains
+
+**Gallery.** Root resolved lazily on first use from the Home tree (no network in a
+constructor). The folder-tree endpoint returns no file sizes, and Galaxy requires an integer
+`size`, so with `fetch_sizes=True` (the default) a listing fetches each file's record to fill
+it in; `fetch_sizes=False` keeps a listing at one call. Under PyFilesystem this was decided per
+call by the requested namespace; fsspec's `ls` carries no such hint, so it is per instance.
+Upload routing: the Gallery is split into media-type sections and a file may only go into
+a folder of the matching section; `upload_fileobj` returns a `Placement`, and a mismatch either raises `GallerySectionMismatch` or, with
+`on_mismatch="reroute"`, lets RSpace file it and reports where it landed.
+
+**Inventory.** Root lists benches (read via `/workbenches/{id}`, since `/containers/{id}`
+answers 422 for a bench) and the sections. A container lists its child containers and
+subsamples plus attachments; a sample lists its subsamples plus attachments; a subsample lists
+its attachments plus a `sample: ` shortcut folder holding the sample's attachments only,
+so the tree has no cycles. Samples are not stored in containers, their subsamples are.
+
+Records carry files in two disjoint places: their own `attachments`, and template-defined
+attachment fields (`SF`) on samples, templates and instruments, each holding at most one file.
+Only `attachment`-type fields are shown, as folders of 0 or 1 file. Uploading into an occupied
+field is refused with `FileExistsError` rather than replacing, because a folder reads as
+something you add to. Subsamples have attachments but no fields (84 live ones checked).
+
+Where a file can go, verified live against RSpace 2.27 and refused client-side before any
+upload where the server would refuse:
+
+| Record | Own attachments | Attachment fields |
+| --- | --- | --- |
+| Sample | read and write | read and write where the template defines one |
+| Subsample, container | read and write | none exist |
+| Instrument | read and write | where the template defines one |
+| Sample template | server refuses | browsable |
+| Bench | server refuses | none exist |
+
+An uploaded attachment becomes a **Gallery file** linked to the record, which is what the web
+interface's "link from Gallery" produces. The alternative the API offers creates a file that
+exists only inside Inventory and is reusable nowhere; `via_gallery=False` keeps it, and the
+deprecated `InventoryAttachmentFilesystem` is pinned to it. Linking is also what a copy within
+RSpace means: `RSpaceFilesystem.copy` from any Gallery-backed source into an Inventory record
+or an ELN field calls the link endpoint instead of downloading and re-uploading, which used to
+leave a permanent duplicate behind.
+
+**Workspace.** The Home tree with the Gallery root and the Templates system
+folder hidden. A document exposes its fields as sub-folders, using the field's own name; the
+server's `ApiDocumentField` already supplies each field's linked `files`, so no HTML parsing is
+needed to list them. Only `text` fields are shown: the form editor offers no other type that
+holds files, every caller of `Field.addMediaFileLink` in rspace-web passes a text field, and the
+legacy `Attachment` type is uncreatable and always empty. The document's entry counts the
+omitted fields in `rspace.hiddenFields`. Uploading puts the bytes in the Gallery and appends a
+`` token to that field; the server rewrites the token into an `attachmentDiv` or an
+`` on save, and unlinking strips exactly those forms with BeautifulSoup so a field id is
+never mistaken for a file id. `makedir` creates folders only. Only the user's own Home tree is
+reachable; shared and group documents are not in it.
+
+The same Gallery file can appear at several paths: under `/gallery` and under every
+field it is linked into. The namespace is a set of overlapping views, not a tree of uniquely
+located files.
+
+## 5. Listings and pagination
+
+Every RSpace listing call already returns `name`, `globalId` and timestamps per child, so each
+branch serves `ls` and `scandir` from one listing call rather than one `info` per child, and
+fills `size`, `created` and `mtime` as epoch seconds. Entries follow fsspec's convention:
+`name` is the full path without a leading slash. The base class implements `ls` itself rather
+than relying on fsspec's `_ls_from_cache` and default `info`, which return a directory's own
+entry among its children and fall back to listing a path to describe it. fsspec's listings
+cache is off by default and cleared on every write.
+
+Section and folder listings are lazy streams that follow the API's `next` links only as far
+as the consumer reads. `page_size` is the fetch batch size, default 100, invisible in the
+namespace. This replaced `page-N` pseudo-folders after seeing them in Galaxy's file browser,
+where they read as data rather than as an implementation detail, and Galaxy paginates
+listings itself (after a full `ls`), so page folders had it paginating over our pagination.
+
+## 6. Errors
+
+RSpace client exceptions are translated once, at every public method of the base class (and
+of every subclass, which `__init_subclass__` wraps), into builtin `OSError` subclasses, which
+is what fsspec's own `exists`, `walk` and `copy`, and Galaxy, understand: a 404 becomes
+`FileNotFoundError`, 401 and 403 `PermissionError`, a lost connection or a 502 to 504
+`ConnectionError`. Every other API refusal, a 500 with a message included, becomes
+`RemoteApiError`, an `OSError` that keeps the server's explanation in its message and the
+original exception as `api_error`. `GallerySectionMismatch` passes through as it is, since its
+message already says what to do.
+
+## 7. Opener
+
+`fsspec.filesystem("rspace", server=...)` and `rspace://host/...` URLs (`fsspec.open`,
+`UPath`) return an `RSpaceFilesystem`, through an `fsspec.specs` entry point, so nothing has to
+be imported first. The URL carries the host and options only. The key is read from
+`RSPACE_API_KEY` and nothing else: a URL is often supplied by someone other than the person
+whose environment it is opened in, so it must not be able to name the variable that leaves the
+machine. User info in the URL is refused. Plain http is accepted for loopback hosts only. The
+client never follows a redirect, because requests keeps the `apiKey` header across a change of
+host.
+
+## 8. Galaxy, the load-bearing consumer
+
+Galaxy pins `rspace-client>=2.6.1,<3` and ships a file source (`galaxy/files/sources/rspace.py`,
+since Galaxy 25.1) that **subclasses the deprecated** `rspace_client.eln.fs.GalleryFilesystem`,
+monkeypatches `ELNClient.upload_file` with a function taking only `(file, folder_id, caption)`,
+and passes it a `FakedNameIO` wrapper that forwards `read` through `__getattr__`. It relies on
+five things no documented API promises:
+
+| Galaxy relies on | Kept by |
+| --- | --- |
+| `upload_file` accepting no extra keyword | the shim sends no `filename=`; only `rspace_client.fs` does |
+| `upload(folder_path, file)` naming the file after the file object | the shim's `upload`, unchanged |
+| bare global-ID segments, from which it builds URIs | the shim, unchanged |
+| `basic.name`, `rspace.name`, ISO `rspace.created`, integer `size` | the shim's `getinfo`, unchanged |
+| a file part that only forwards `read` being accepted | `_post_multipart` wraps it, see below |
+
+The last row is new with requests 2.34 (pinned by Galaxy 26.1): requests now checks that a
+multipart file part defines `read` on its class, which `FakedNameIO` does not, so Galaxy's
+export to RSpace fails with a `TypeError` whatever rspace-client is installed. The client now
+wraps such an object in one that defines `read`, which fixes the shipped plugin from the
+client side.
+
+All five are pinned by `rspace_client/tests/galaxy_plugin_contract_test.py`, which copies
+Galaxy's wrapper classes so the suite runs without Galaxy installed. **A failure there is a
+release blocker, not a test to update.** The shims (`rspace_client.eln.fs`, which Galaxy
+imports, and `rspace_client.inv.attachment_fs`, which it does not but 2.x users may) are the
+2.7 PyFilesystem classes, with their section-routing helpers now imported
+from `rspace_client.fs.gallery` and a `DeprecationWarning` on construction. They cannot be
+removed on a schedule: removal follows a Galaxy release that no longer imports them, no
+earlier than 3.0, together with the `fs` dependency and the `setuptools` pin.
+
+Galaxy discovers file sources by walking its own package, so exposing `/inventory` and
+`/workspace` to Galaxy users needs a pull request to `galaxyproject/galaxy`:
+`tools/galaxy/rspace_fsspec_source.py` is the proposed file source, built on Galaxy's
+`FsspecFilesSource`. It also needs a file source template type, since Galaxy's template catalog
+rejects unknown types, and a raised `rspace-client` pin. Pointing the existing `rspace` type at
+it with `mounts=gallery` and `path_style="id"` would keep saved URIs working and retire the
+deprecated import. That pull request depends on a released `rspace-client` that contains this
+package.
diff --git a/docs/usage-guide.md b/docs/usage-guide.md
index 4f2b9a4..2e4bce4 100644
--- a/docs/usage-guide.md
+++ b/docs/usage-guide.md
@@ -9,7 +9,7 @@ bash> export RSPACE_URL=https://myrspace.com
bash> export RSPACE_API_KEY=abcdefgh...
```
-- [Using the library as a PyFilesystem implementation](#using-the-rspace_client-library-as-pyfilesystem-implementation)
+- [Using RSpace as a filesystem (fsspec)](#using-rspace-as-a-filesystem-fsspec)
- [A basic query to list documents](#a-basic-query-to-list-documents)
- [Iterating over pages of results](#iterating-over-pages-of-results)
- [Searching](#searching)
@@ -24,92 +24,135 @@ bash> export RSPACE_API_KEY=abcdefgh...
- [Forms](#forms)
- [Export](#export)
-## Using the rspace_client library as PyFilesystem implementation
+## Using RSpace as a filesystem (fsspec)
-Starting with rspace-client 2.6.0 rspace-client the library partially implements
-[PyFilesystem](https://docs.pyfilesystem.org/en/latest/index.html) API for accessing
-RSpace Gallery files and RSpace Inventory attachments.
+`rspace_client.fs` presents RSpace as an [fsspec](https://filesystem-spec.readthedocs.io)
+filesystem, the same interface pandas, pyarrow, dask, zarr, `universal_pathlib` and Galaxy
+use for remote storage. One mount holds three branches:
-To start, export the URL and API key into environment variables (as explained before).
+```
+gallery/ Gallery folders and files
+inventory/ benches and the sections Containers, Samples, Templates; every record is a
+ folder holding its child records and its attachments
+workspace/ ELN folders, notebooks and documents; a document is a folder of its text
+ fields, a field is a folder of the files linked into it
+```
+
+A directory is anything that contains files; a file is a downloadable blob. Paths are the
+records' own names, without a leading slash (`gallery/Images/microscope.png`). Every entry
+returned by `ls()` or `info()` is a dict with `name`, `type`, `size`, `mtime`, `created`,
+`globalId` and `rspace` (the raw RSpace record).
-To access the Gallery files construct a `GalleryFilesystem` object.
```python
-from rspace_client.eln import fs
+from rspace_client.fs import RSpaceFilesystem, print_tree
-# create rspace_fs object which supports generic fs methods (listdir, getinfo etc.)
-rspace_gallery_fs = fs.GalleryFilesystem(os.getenv("RSPACE_URL"), os.getenv("RSPACE_API_KEY"))
+fs = RSpaceFilesystem(os.getenv("RSPACE_URL"), os.getenv("RSPACE_API_KEY"))
-content = rspace_gallery_fs.listdir("/")
-print(content)
-for globalId in content:
- print(rspace_gallery_fs.getinfo(globalId).raw)
+fs.ls("", detail=False) # ['gallery', 'inventory', 'workspace']
+fs.ls("gallery/Images", detail=False) # ['gallery/Images/microscope.png', ...]
+fs.info("gallery/Images/microscope.png") # {'name': ..., 'type': 'file', 'size': 1096, 'globalId': 'GL100', ...}
+data = fs.cat_file("gallery/Images/microscope.png")
+fs.get_file("inventory/Containers/Freezer -80/manual.pdf", "manual.pdf")
+print_tree(fs, "workspace/Project Alpha", max_depth=3)
```
-#### Uploading into Gallery folders and the media-type section rule
-
-The RSpace Gallery is split into media-type sections (Images, Documents,
-Chemistry, ...) and a file can only be placed in a folder whose section matches
-the file's media type. Uploading, say, a PDF into a folder that lives in the
-Images section is rejected by the server.
-
-By default `upload` turns that rejection into a clear `GallerySectionMismatch`
-that names the folder's section, rather than an opaque API error:
+With `RSPACE_API_KEY` in the environment the filesystem also opens from a URL. The URL
+carries the host and options only, never the key:
```python
-from rspace_client.eln.fs import GallerySectionMismatch
+import fsspec
+from upath import UPath
-try:
- rspace_gallery_fs.upload("/GF123", file_obj) # GF123 is in the Images section
-except GallerySectionMismatch as e:
- print(e) # explains the section clash
- print(e.folder_section) # e.g. "Images"
+fs = fsspec.filesystem("rspace", server="https://my.rspace.host")
+with fsspec.open("rspace://my.rspace.host/gallery/Documents/data.csv") as f:
+ ...
+for entry in UPath("rspace://my.rspace.host/inventory/Samples").iterdir():
+ print(entry.name)
```
-Alternatively, opt in to automatic rerouting. On a mismatch the file is placed
-in the correct section's inbox instead of failing. Set the policy once when
-constructing the filesystem (it then applies to every write, including generic
-PyFilesystem operations), or override it per call:
+Plain http is accepted for loopback hosts only (`rspace://localhost:8080/...?scheme=http`).
-```python
-# filesystem-wide default
-rspace_gallery_fs = fs.GalleryFilesystem(url, api_key, on_mismatch="reroute")
+### Writing
-# or per upload
-placement = rspace_gallery_fs.upload("/GF123", file_obj, on_mismatch="reroute")
-```
-
-`upload` returns a `Placement` telling you where the file actually landed:
+Everything is read-only by default. `writable=True` allows uploads and folder creation,
+`allow_delete=True` allows removals; the two are separate on purpose.
```python
-placement.rerouted # True if it did not go in the requested folder
-placement.section # e.g. "Documents"
-placement.path # e.g. "Gallery/Documents/Api Inbox"
-placement.file_global_id # e.g. "GL999"
+fs = RSpaceFilesystem(url, api_key, writable=True, allow_delete=True)
+
+with fs.open("gallery/Documents/notes.txt", "wb") as f: # uploads when closed
+ f.write(b"...")
+fs.put_file("results.csv", "workspace/Project Alpha/Alpha protocol/Results/results.csv")
+fs.mkdir("gallery/Images/2026-10 run")
+fs.rm_file("inventory/Samples/Plasmid pUC19/old_gel.png") # deletes the attachment
+fs.rm_file("workspace/Project Alpha/Alpha protocol/Results/old.csv") # unlinks; the Gallery file stays
```
-Note that rerouting places the file in the section's inbox, not a subfolder
-matching the one you requested.
+What a write means depends on the branch: a file put into a Gallery folder is a Gallery
+upload; into an Inventory record it is uploaded to the Gallery and attached to the record;
+into a document field it is uploaded to the Gallery and linked into that field. A copy from
+any Gallery-backed file into Inventory or a document field (`fs.cp_file`) links the existing
+file instead of moving bytes. Transactions defer uploads until the block ends:
-To access Inventory attachments construct a `InventoryAttachmentFilesystem` object.
```python
-from rspace_client.inv.attachment_fs import InventoryAttachmentFilesystem
+with fs.transaction:
+ with fs.open("gallery/Documents/a.txt", "wb") as f: f.write(b"a")
+ with fs.open("gallery/Documents/b.txt", "wb") as f: f.write(b"b")
+# both uploaded here; neither if the block raised
+```
+
+A signed ELN document is locked and its fields are listed with a `(signed)` suffix. An
+Inventory attachment field holds one file and refuses an upload while occupied. A bench or
+sample template takes no attachments, and the Gallery API cannot delete Gallery files; each
+refusal says so before anything is sent.
+
+Folders are not created on the way: a file can only be written into a folder that already
+exists, and `fs.makedirs` and `fs.mkdir(..., create_parents=True)` create only the last
+segment of the path, unlike `os.makedirs`. Writing `gallery/Documents/run-42/data.csv` when
+`run-42` does not exist raises `FileNotFoundError` for `Documents/run-42`; create each level
+with `fs.mkdir` first. (Only Gallery and Workspace folders can be created at all; files in the
+Workspace go into a document's text field, and Inventory records are not made from paths.)
+
+#### Names, duplicates and global IDs
+
+Where two records in a folder share a name, the second and later ones are listed with their
+global ID before the extension (`data [GL112].csv`), and a name that would itself read as an
+address gets its real ID the same way. A segment carrying a global ID always resolves to that
+record, whatever the style, and survives a rename. For a file the folder above it must really
+hold it: `Results/GL112` is refused with `FileNotFoundError` when GL112 is not linked into
+`Results`, rather than reaching a file that lives somewhere else. `path_style="labelled"` lists every
+segment as `Name [GID]`; `path_style="id"` lists bare global IDs.
+
+#### Gallery sections
+
+The Gallery is split into media-type sections (Images, Documents, Chemistry, ...) and a file
+can only go into a folder of the matching section. A mismatch raises
+`GallerySectionMismatch`, which names the folder's section; `on_mismatch="reroute"` (on the
+filesystem or per `upload_fileobj` call) lets RSpace file it in the right section instead
+and the returned `Placement` says where it landed.
-rspace_inv_fs = InventoryAttachmentFilesystem(os.getenv("RSPACE_URL"), os.getenv("RSPACE_API_KEY"))
+```python
+placement = fs.upload_fileobj("gallery/Images/report.pdf", file_obj, on_mismatch="reroute")
+placement.rerouted, placement.section, placement.path, placement.file_global_id
+```
-# list the attachments that an existing record has, in this case a subsample
-rspace_inv_fs.listdir('/SS123')
+#### Galaxy
-# get the metadata for a particular attachment
-rspace_inv_fs.getinfo("IF123")
+Galaxy's file sources are built on fsspec. `tools/galaxy/rspace_fsspec_source.py` is the
+reference plugin for `RSpaceFilesystem`; the shipped Galaxy plugin still uses the
+deprecated classes below. Because folders are not created on the way (see
+[Writing](#writing)), a Galaxy export into a folder typed into the export form's "enter
+directory name" box fails with "not found" unless that folder already exists in RSpace;
+choose an existing folder, or create it first.
-# or download it
-file_obj = BytesIO()
-rspace_inv_fs.download('/IF123', file_obj)
+### Deprecated: the PyFilesystem classes
-# or attach a new file
-file_obj_2 = BytesIO(b'test file content')
-rspace_inv_fs.upload('/SS123', file_obj_2)
-```
+`rspace_client.eln.fs.GalleryFilesystem` and
+`rspace_client.inv.attachment_fs.InventoryAttachmentFilesystem` implement the unmaintained
+[PyFilesystem2](https://github.com/PyFilesystem/pyfilesystem2) API with global IDs as path
+segments (`/GF123`, `/IF123`). They are kept unchanged for the Galaxy plugin that imports
+them, emit a `DeprecationWarning`, and will be removed in rspace-client 3.0 together with
+the `fs` dependency. New code should use `rspace_client.fs`.
## A basic query to list documents
diff --git a/pyproject.toml b/pyproject.toml
index 65b7b84..c790eff 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -24,14 +24,20 @@ exclude = ["examples/*", "rspace_client/tests/*"]
python = "^3.9"
requests = "^2.25.1"
beautifulsoup4 = "^4.9.3"
-fs = "^2.4.16"
-setuptools = "<82"
+fs = "^2.4.16" # deprecated PyFilesystem2 classes only (rspace_client.eln.fs, rspace_client.inv.attachment_fs); removed in 3.0
+setuptools = "<82" # fs 2.4.16 imports pkg_resources; goes with fs
+fsspec = ">=2024.3.0" # rspace_client.fs
+
+# lets fsspec.filesystem("rspace") and rspace:// URLs work before rspace_client.fs is imported
+[tool.poetry.plugins."fsspec.specs"]
+rspace = "rspace_client.fs:RSpaceFilesystem"
[tool.poetry.group.dev.dependencies]
python-dotenv = "^1.1.1"
black = "^21.6b0"
pytest = "^8.0.0"
responses = "^0.26.2"
+universal-pathlib = ">=0.2"
[build-system]
requires = ["poetry-core>=1.0.0"]
diff --git a/rspace_client/client_base.py b/rspace_client/client_base.py
index 45e4e03..8a12bc4 100644
--- a/rspace_client/client_base.py
+++ b/rspace_client/client_base.py
@@ -1,3 +1,4 @@
+import inspect
import logging
import re
import requests
@@ -25,6 +26,35 @@
REDIRECT_STATUSES = (301, 302, 303, 307, 308)
+class _ReadForwarder:
+ """
+ Gives requests a file object it recognises. Since requests 2.34, a multipart file part
+ must define ``read`` on its class: on Python 3.12+ the check skips ``__getattr__``, so
+ a wrapper that forwards ``read`` dynamically (Galaxy's ``FakedNameIO``, which Galaxy's
+ RSpace file source passes to ``ELNClient.upload_file``) is refused with a TypeError.
+ """
+
+ def __init__(self, file):
+ self._file = file
+ self.name = getattr(file, "name", None) # requests names the part after it
+
+ def read(self, *args):
+ return self._file.read(*args)
+
+
+def _readable_part(part):
+ """A multipart file part, with a file object that only reads dynamically wrapped."""
+ if isinstance(part, (tuple, list)) and len(part) >= 2:
+ return (part[0], _readable_part(part[1])) + tuple(part[2:])
+ if (
+ not isinstance(part, (str, bytes, bytearray))
+ and inspect.getattr_static(part, "read", None) is None
+ and callable(getattr(part, "read", None))
+ ):
+ return _ReadForwarder(part)
+ return part
+
+
class _RSpaceRetry(Retry):
"""
Retry policy that additionally retries POST requests, but only on
@@ -186,7 +216,20 @@ def _get_error_detail(response):
return ClientBase._get_formated_error_message(response.json())
except ValueError:
pass
- return "error message: {}".format(response.text)
+ text = response.text or ""
+ if text.lstrip()[:15].lower().startswith(("", " ", text, flags=re.I | re.S)
+ visible = " ".join(re.sub(r"<[^>]+>", " ", visible).split())
+ if len(visible) <= 200:
+ return "error message: {}".format(visible or "an empty HTML page")
+ title = re.search(r"]*>(.*?)", text, re.I | re.S)
+ title = " ".join(title.group(1).split()) if title else ""
+ return "error message: the server answered with an HTML error page{}".format(
+ f" ({title})" if title else "")
+ return "error message: {}".format(text)
@staticmethod
def _check_status(response):
@@ -296,6 +339,8 @@ def _post_multipart(self, endpoint, files=None, data=None):
url = endpoint
if not self._is_absolute_url(endpoint):
url = self._get_api_url() + endpoint
+ if files:
+ files = {key: _readable_part(part) for key, part in files.items()}
try:
response = self._session.post(
url,
diff --git a/rspace_client/eln/fs.py b/rspace_client/eln/fs.py
index 060c098..0f8cbca 100644
--- a/rspace_client/eln/fs.py
+++ b/rspace_client/eln/fs.py
@@ -1,5 +1,5 @@
import logging
-from dataclasses import dataclass
+import warnings
from fs.base import FS
from rspace_client.eln import eln
from rspace_client.client_base import ClientBase
@@ -14,61 +14,11 @@
logger = logging.getLogger(__name__)
-# Accepted values for the GalleryFilesystem ``on_mismatch`` policy.
-ON_MISMATCH_RAISE = "raise"
-ON_MISMATCH_REROUTE = "reroute"
-_ON_MISMATCH_VALUES = (ON_MISMATCH_RAISE, ON_MISMATCH_REROUTE)
-
-# The RSpace Gallery's media-type sections. "Miscellaneous" is the catch-all
-# that accepts any file not matching a more specific section; the others accept
-# only their listed extensions (including "Documents", which is a fixed set, not
-# a catch-all).
-MISCELLANEOUS_SECTION = "Miscellaneous"
-
-# Best-effort mapping of file extension to the Gallery section RSpace classifies
-# it into, taken from the Gallery documentation. This mirrors the server's own
-# classification but is NOT authoritative: it is used only to phrase error
-# messages, never to decide whether an upload is allowed. Any extension not
-# listed falls through to "Miscellaneous", matching the server default.
-_SECTION_BY_EXTENSION = {
- # Images
- "png": "Images", "jpg": "Images", "jpeg": "Images", "gif": "Images",
- "bmp": "Images", "tif": "Images", "tiff": "Images",
- # Audios
- "mp3": "Audios", "wav": "Audios", "wma": "Audios", "aac": "Audios",
- "ogg": "Audios",
- # Videos
- "mp4": "Videos", "mov": "Videos", "hdmov": "Videos", "m4v": "Videos",
- "wmv": "Videos", "avi": "Videos", "mpg": "Videos", "mpeg": "Videos",
- "flv": "Videos", "3gp": "Videos",
- # Documents (a fixed set, NOT a catch-all)
- "doc": "Documents", "docx": "Documents", "rtf": "Documents",
- "pdf": "Documents", "odt": "Documents", "ods": "Documents",
- "odp": "Documents", "txt": "Documents", "ppt": "Documents",
- "pptx": "Documents", "xls": "Documents", "xlsx": "Documents",
- "csv": "Documents", "pps": "Documents", "md": "Documents",
- # Chemistry (documented subset; the server accepts more)
- "skc": "Chemistry", "mrv": "Chemistry", "cxsmiles": "Chemistry",
- "cxsmarts": "Chemistry", "cdx": "Chemistry", "cdxml": "Chemistry",
- "csrdf": "Chemistry", "cml": "Chemistry",
-}
-
-
-def classify_media_section(filename: Optional[Text]) -> Optional[Text]:
- """
- Best-effort guess of the Gallery section for a filename, mirroring RSpace's
- server-side classification. Returns a section name (e.g. "Images"),
- "Miscellaneous" as the catch-all when the extension is not one of a specific
- section's types, or None when no filename/extension is available to guess
- from.
-
- The server remains the authority on placement; this is only used to make
- error messages and log lines more helpful.
- """
- if not filename or "." not in filename:
- return None
- ext = filename.rsplit(".", 1)[-1].lower()
- return _SECTION_BY_EXTENSION.get(ext, MISCELLANEOUS_SECTION)
+# The section policy, the classification table, the exception and the Placement record
+# are shared with the fsspec filesystem that replaces this module.
+from rspace_client.fs.gallery import (MISCELLANEOUS_SECTION, ON_MISMATCH_RAISE, ON_MISMATCH_REROUTE, # noqa: E402,F401
+ GallerySectionMismatch, Placement, check_policy, classify_media_section, folder_section,
+ mismatch_message, placement)
def _filename_for(file: BinaryIO, options: Mapping[Text, Any]) -> Optional[Text]:
@@ -81,37 +31,15 @@ def _filename_for(file: BinaryIO, options: Mapping[Text, Any]) -> Optional[Text]
return name if isinstance(name, str) else None
-class GallerySectionMismatch(ClientBase.ApiError):
- """
- Raised when a file could not be uploaded into the requested Gallery folder
- because that folder belongs to a media-type section that does not accept the
- file. Carries the folder's section and, when it could be guessed, the file's
- media type, so callers can react programmatically as well as read the message.
- """
-
- def __init__(self, message, *, folder_section=None, folder_global_id=None,
- file_media_type=None, response_status_code=None):
- super().__init__(message, response_status_code=response_status_code)
- self.folder_section = folder_section
- self.folder_global_id = folder_global_id
- self.file_media_type = file_media_type
-
-
-@dataclass
-class Placement:
- """Where an uploaded file actually ended up in the Gallery.
-
- Returned by :meth:`GalleryFilesystem.upload`. When ``rerouted`` is True the
- file did not land in the folder named by ``requested_path`` (its section did
- not accept the file) and was placed in the correct section instead;
- ``path`` reports the human-readable location it ended up in.
- """
- file_global_id: Optional[Text]
- folder_global_id: Optional[Text]
- section: Optional[Text]
- path: Text
- rerouted: bool
- requested_path: Optional[Text] = None
+def _with_parsable_created(info: Mapping[Text, Any]) -> Mapping[Text, Any]:
+ """RSpace dates end in ``Z`` for UTC. Galaxy's ``rspace`` file source parses ``created``
+ with ``datetime.fromisoformat``, which rejects ``Z`` before Python 3.11, so on 3.10 every
+ file listing failed. Spell the same instant as ``+00:00``; the rest of the record is
+ passed through untouched."""
+ created = info.get("created")
+ if isinstance(created, str) and created.endswith("Z"):
+ return {**info, "created": created[:-1] + "+00:00"}
+ return info
def is_folder(path):
@@ -127,6 +55,10 @@ def __init__(self, obj, *args, **kwargs) -> None:
class GalleryFilesystem(FS):
def __init__(self, server: str, api_key: str, on_mismatch: str = ON_MISMATCH_RAISE) -> None:
+ warnings.warn(
+ "rspace_client.eln.fs.GalleryFilesystem (PyFilesystem2) is deprecated and will be removed "
+ "in rspace-client 3.0; use rspace_client.fs.GalleryFilesystem or RSpaceFilesystem (fsspec)",
+ DeprecationWarning, stacklevel=2)
"""
:param server: RSpace server URL
:param api_key: RSpace API key
@@ -139,11 +71,7 @@ def __init__(self, server: str, api_key: str, on_mismatch: str = ON_MISMATCH_RAI
:meth:`upload` calls may override it.
"""
super(GalleryFilesystem, self).__init__()
- if on_mismatch not in _ON_MISMATCH_VALUES:
- raise ValueError(
- "on_mismatch must be one of {}".format(_ON_MISMATCH_VALUES)
- )
- self.on_mismatch = on_mismatch
+ self.on_mismatch = check_policy(on_mismatch)
self.eln_client = eln.ELNClient(server, api_key)
self.gallery_id = next(file['id'] for file in self.eln_client.list_folder_tree()['records'] if file['name'] == 'Gallery')
@@ -165,7 +93,7 @@ def getinfo(self, path, namespaces=None) -> Info:
"size": info.get('size', 0),
"type": is_folder(path) and "1" or "2",
},
- "rspace": info,
+ "rspace": _with_parsable_created(info),
})
def listdir(self, path: Text) -> List[Text]:
@@ -232,63 +160,6 @@ def download(self, path: Text, file: BinaryIO, chunk_size: Optional[int] = None,
else:
self.eln_client.download_file(path_to_id(path), file)
- def _folder_section(self, folder_id: Text) -> Optional[Text]:
- """The Gallery section (mediaType) a folder belongs to, or None if it
- cannot be determined. Best-effort: never raises, so it cannot mask the
- real outcome of an upload."""
- try:
- return self.eln_client.get_folder(folder_id).get("mediaType")
- except Exception:
- return None
-
- def _human_path(self, folder: Mapping[Text, Any]) -> Text:
- """Best-effort readable path for a folder, e.g. 'Gallery/Documents/Api
- Inbox'. Uses the API's pathToRootFolder when present, otherwise falls
- back to the section and folder name."""
- trail = folder.get("pathToRootFolder")
- if isinstance(trail, list) and trail:
- names = [f.get("name") for f in trail if f.get("name")]
- if names:
- return "/".join(names)
- parts = ["Gallery"]
- if folder.get("mediaType"):
- parts.append(folder["mediaType"])
- if folder.get("name"):
- parts.append(folder["name"])
- return "/".join(parts)
-
- def _placement(self, response: Any, requested_path: Optional[Text],
- rerouted: bool) -> Placement:
- """Build a Placement from an upload response, resolving the parent
- folder for section/path feedback where possible.
-
- Tolerates a response that is not a dict (e.g. None): some callers wrap
- or replace ``eln_client.upload_file`` and discard its return value, so
- Placement construction must never crash a successful upload.
- """
- if not isinstance(response, dict):
- response = {}
- parent_id = response.get("parentFolderId")
- section = None
- folder_global_id = None
- path = "Gallery"
- if parent_id is not None:
- try:
- folder = self.eln_client.get_folder(parent_id)
- section = folder.get("mediaType")
- folder_global_id = folder.get("globalId")
- path = self._human_path(folder)
- except Exception:
- pass
- return Placement(
- file_global_id=response.get("globalId"),
- folder_global_id=folder_global_id,
- section=section,
- path=path,
- rerouted=rerouted,
- requested_path=requested_path,
- )
-
def upload(self, path: Text, file: BinaryIO, chunk_size: Optional[int] = None,
on_mismatch: Optional[str] = None, **options: Any) -> Placement:
"""
@@ -309,19 +180,13 @@ def upload(self, path: Text, file: BinaryIO, chunk_size: Optional[int] = None,
section (``"raise"``) or places the file in the correct section's inbox
and returns a Placement with ``rerouted=True`` (``"reroute"``).
"""
- policy = on_mismatch if on_mismatch is not None else self.on_mismatch
- if policy not in _ON_MISMATCH_VALUES:
- raise ValueError(
- "on_mismatch must be one of {}".format(_ON_MISMATCH_VALUES)
- )
+ policy = check_policy(self.on_mismatch if on_mismatch is None else on_mismatch)
folder_id = path_to_id(path) if path else None
try:
response = self.eln_client.upload_file(file, folder_id)
- return self._placement(response, requested_path=path or None, rerouted=False)
+ return placement(self.eln_client, response, requested_path=path or None, rerouted=False)
except ClientBase.ApiError as err:
- if folder_id is None:
- raise
- section = self._folder_section(folder_id)
+ section = None if folder_id is None else folder_section(self.eln_client, folder_id)
if section is None:
raise
filename = _filename_for(file, options)
@@ -333,39 +198,14 @@ def upload(self, path: Text, file: BinaryIO, chunk_size: Optional[int] = None,
except (AttributeError, OSError, ValueError):
pass
response = self.eln_client.upload_file(file, None)
- placement = self._placement(response, requested_path=path, rerouted=True)
+ placed = placement(self.eln_client, response, requested_path=path, rerouted=True)
logger.info(
- "RSpace Gallery: %s could not go in %s (section '%s'); "
- "placed in %s instead",
- "'{}'".format(filename) if filename else "file",
- path, section, placement.path,
- )
- return placement
+ "RSpace Gallery: %s could not go in %s (section '%s'); placed in %s instead",
+ "'{}'".format(filename) if filename else "file", path, section, placed.path)
+ return placed
- named = "'{}'".format(filename) if filename else "the file"
- message = (
- "Could not upload {named} to Gallery folder {path}. That folder "
- "is in the '{section}' section, which only accepts {section} "
- "files".format(named=named, path=path, section=section)
- )
- if guessed and guessed != section:
- if guessed == MISCELLANEOUS_SECTION:
- message += (
- ", but {named} does not match a specialised section and "
- "belongs in 'Miscellaneous'".format(named=named)
- )
- else:
- message += ", but {named} looks like a '{guessed}' file".format(
- named=named, guessed=guessed
- )
- message += (
- ". Upload it to a folder in the matching section, omit the "
- "folder path to let RSpace place it in the correct section "
- "automatically, or construct the filesystem with "
- "on_mismatch='reroute'. Original API error: {err}".format(err=err)
- )
raise GallerySectionMismatch(
- message,
+ mismatch_message("'{}'".format(filename) if filename else "the file", path, section, guessed, err),
folder_section=section,
folder_global_id="GF" + str(folder_id),
file_media_type=guessed,
diff --git a/rspace_client/fs/__init__.py b/rspace_client/fs/__init__.py
new file mode 100644
index 0000000..ac71c02
--- /dev/null
+++ b/rspace_client/fs/__init__.py
@@ -0,0 +1,48 @@
+"""
+fsspec (https://filesystem-spec.readthedocs.io) implementations for RSpace.
+
+ from rspace_client.fs import GalleryFilesystem, InventoryFilesystem, WorkspaceFilesystem
+ from rspace_client.fs import RSpaceFilesystem # one mount: gallery/, inventory/, workspace/
+
+ import fsspec # key from RSPACE_API_KEY:
+ fsspec.filesystem("rspace", server="https://my.rspace.host")
+ fsspec.open("rspace://my.rspace.host/gallery/Images/microscope.png").open().read()
+
+All filesystems are read-only by default. Pass ``writable=True`` to allow uploads and
+folder creation, and ``allow_delete=True`` to allow removals. ``mounts=`` narrows what
+``RSpaceFilesystem`` exposes, for a caller that only wants part of RSpace.
+
+A path segment is the record's own name (``gallery/Images/microscope.png``). Where two
+records in a folder share a name, the later ones carry their global ID before the extension
+(``data [GL112].csv``). A segment written as a global ID always resolves, whatever style is
+in force (a file only under the folder, field or record that really holds it), and ``path_style="labelled"`` or ``"id"`` put one on every segment, which is what
+you want if a stored path has to survive a rename. The full RSpace record for any entry is
+under the ``rspace`` key of its info dict.
+"""
+import fsspec
+
+from .base import RSpaceFSBase, ReadOnlyError, RemoteApiError, make_entry
+from .gallery import GalleryFilesystem, GallerySectionMismatch, Placement
+from .inventory import InventoryFilesystem
+from .workspace import WorkspaceFilesystem
+from .rspace import RSpaceFilesystem
+from .tree import format_tree, print_tree
+
+# also declared as an ``fsspec.specs`` entry point in pyproject.toml, so that
+# ``fsspec.filesystem("rspace")`` works before this package was imported
+fsspec.register_implementation("rspace", RSpaceFilesystem, clobber=True)
+
+__all__ = [
+ "RSpaceFSBase",
+ "ReadOnlyError",
+ "RemoteApiError",
+ "make_entry",
+ "GalleryFilesystem",
+ "GallerySectionMismatch",
+ "Placement",
+ "InventoryFilesystem",
+ "WorkspaceFilesystem",
+ "RSpaceFilesystem",
+ "format_tree",
+ "print_tree",
+]
diff --git a/rspace_client/fs/base.py b/rspace_client/fs/base.py
new file mode 100644
index 0000000..356d4a7
--- /dev/null
+++ b/rspace_client/fs/base.py
@@ -0,0 +1,771 @@
+"""
+Shared base class for the RSpace fsspec filesystems.
+
+Provides, so that every branch (Gallery, Inventory, Workspace) behaves the same:
+
+* a path style: ``name`` shows a record's own name, ``labelled`` shows ``Name [GID]``
+ and ``id`` shows bare global IDs. A segment carrying a global ID resolves under
+ every style, in one call; a *bare name* resolves only under ``name``, through
+ ``_lookup``, which matches it against the parent's listing;
+* a write posture: read-only by default, ``writable=True`` enables uploads and
+ folder creation, ``allow_delete=True`` enables removals;
+* ``entry`` dicts in fsspec's shape (``name`` is the full path without a leading
+ slash, ``type`` is ``file`` or ``directory``, ``size``, ``mtime`` and ``created`` are
+ epoch seconds) plus ``globalId``, ``rspace`` (the raw RSpace record) and, where RSpace
+ tells us, ``writable``;
+* ``ls``/``scandir``, which subclasses feed from a single listing call via ``_children``
+ instead of one ``info`` request per child, and which keep the segments in a listing
+ unique and self-addressing even when two records share a name (see ``_listing``);
+* ``stream_pages``, which turns any paginated RSpace response into a lazy stream of
+ records by following its ``next`` links only as far as the consumer reads;
+* file I/O: reads buffer the whole file (RSpace serves no byte ranges), writes buffer
+ until the file is closed or committed and upload in one request, which is what makes
+ fsspec transactions work;
+* error mapping: RSpace client exceptions become builtin ``OSError`` subclasses at every
+ public method, so ``exists``, ``walk``, ``copy`` and Galaxy behave;
+* a ``glob`` that understands the ``/**`` pattern Galaxy's search sends, with
+ the escaping Galaxy applies, so that a segment carrying ``[GL83]`` can be found.
+"""
+from __future__ import annotations
+
+import errno
+import functools
+import io
+import os
+import re
+from dataclasses import dataclass
+from datetime import datetime, timezone
+from typing import (Any, BinaryIO, Callable, Dict, Iterable, Iterator, List, Mapping, Optional,
+ Tuple)
+
+from fsspec.spec import AbstractBufferedFile, AbstractFileSystem
+
+from rspace_client.client_base import ClientBase
+from rspace_client.exceptions import ApiError, AuthenticationError, RSpaceConnectionError
+
+from . import paths
+
+
+# ------------------------------------------------------------------ errors
+#
+# fsspec callers (``exists``, ``walk``, ``copy``, Galaxy's file source) expect ``OSError``
+# subclasses: ``FileNotFoundError`` for a missing path, ``PermissionError`` for a refusal,
+# ``FileExistsError`` for a clash. The RSpace clients raise their own exceptions, so every
+# public filesystem method translates them at the boundary, once, here.
+
+
+class RemoteApiError(OSError):
+ """An RSpace API refusal that is neither "not found" nor "not allowed": a wrong Gallery
+ section (422), a refused link (409), a bad request (400). Carries the server's own
+ explanation in ``str()`` and the original exception as ``api_error``."""
+
+ def __init__(self, path: str, exc: Exception) -> None:
+ super().__init__(errno.EIO, f"{path!r}: {exc}", path)
+ self.api_error = exc
+ self.status = getattr(exc, "response_status_code", None)
+
+
+class ReadOnlyError(PermissionError):
+ """A write was refused because of the filesystem's posture, not by the server."""
+
+
+def os_error_for(exc: Exception, path: str) -> Optional[OSError]:
+ """The ``OSError`` that stands for a client exception, or None to re-raise as it is."""
+ if getattr(exc, "fs_passthrough", False): # e.g. GallerySectionMismatch: already explained
+ return None
+ if isinstance(exc, RSpaceConnectionError):
+ return ConnectionError(f"{path!r}: {exc}")
+ if isinstance(exc, AuthenticationError):
+ return PermissionError(errno.EACCES, f"{path!r}: {exc}", path)
+ status = getattr(exc, "response_status_code", None)
+ if status == 404:
+ return FileNotFoundError(errno.ENOENT, f"{path!r}: {exc}", path)
+ if status in (401, 403):
+ return PermissionError(errno.EACCES, f"{path!r}: {exc}", path)
+ if status in (502, 503, 504): # the server or a proxy in front of it is not answering
+ return ConnectionError(f"{path!r}: {exc}")
+ if isinstance(exc, ApiError):
+ # a 500 included: RSpace answers one for a request it understood but could not
+ # carry out (an invalid image, say), and the message explains why
+ return RemoteApiError(path, exc)
+ return None
+
+
+#: Public methods whose first argument is a path and which may call the server.
+_TRANSLATED_METHODS = ("ls", "scandir", "info", "cat_file", "get_file", "put_file", "_open",
+ "download_fileobj", "upload_fileobj", "mkdir", "makedirs", "rmdir",
+ "rm_file", "link", "glob", "cp_file", "mv", "created", "modified")
+
+
+_CLIENT_ERRORS = (ApiError, AuthenticationError, RSpaceConnectionError)
+
+
+def _reraise(exc: Exception, path: str):
+ translated = os_error_for(exc, path)
+ if translated is None:
+ raise exc
+ raise translated from exc
+
+
+def _translating_iterator(iterator: Iterator, path: str) -> Iterator:
+ try:
+ yield from iterator
+ except _CLIENT_ERRORS as exc:
+ _reraise(exc, path)
+
+
+def translate_api_errors(method):
+ """Wrap a filesystem method so RSpace client errors surface as ``OSError``."""
+ if getattr(method, "_translates_api_errors", False):
+ return method
+
+ @functools.wraps(method)
+ def wrapper(self, path, *args, **kwargs):
+ try:
+ result = method(self, path, *args, **kwargs)
+ except _CLIENT_ERRORS as exc:
+ _reraise(exc, str(path))
+ # a lazy listing fetches further pages while it is consumed; a file handle is
+ # iterable too but must be returned as it is
+ if isinstance(result, Iterator) and not hasattr(result, "read"):
+ return _translating_iterator(result, str(path))
+ return result
+
+ wrapper._translates_api_errors = True
+ return wrapper
+
+
+def translate_class(cls) -> None:
+ """Apply ``translate_api_errors`` to the public methods ``cls`` itself defines."""
+ for name in _TRANSLATED_METHODS:
+ if name in cls.__dict__:
+ setattr(cls, name, translate_api_errors(cls.__dict__[name]))
+
+
+def writes(method):
+ """Mark a method that changes the server: it is refused unless ``writable=True``."""
+ @functools.wraps(method)
+ def wrapper(self, path, *args, **kwargs):
+ self._require_writable(path)
+ return method(self, path, *args, **kwargs)
+ return wrapper
+
+
+def deletes(method):
+ """Mark a method that removes something: it is refused unless ``allow_delete=True``."""
+ @functools.wraps(method)
+ def wrapper(self, path, *args, **kwargs):
+ self._require_delete(path)
+ return method(self, path, *args, **kwargs)
+ return wrapper
+
+
+# ------------------------------------------------------------------ entries
+
+
+def to_epoch(value: Any) -> Optional[float]:
+ """Convert an RSpace timestamp (ISO-8601 string, or epoch millis/seconds) to epoch seconds."""
+ if value is None or value == "":
+ return None
+ if isinstance(value, (int, float)):
+ return float(value) / 1000.0 if value > 1e11 else float(value)
+ text = str(value).strip()
+ if text.endswith("Z"):
+ text = text[:-1] + "+00:00"
+ try:
+ parsed = datetime.fromisoformat(text)
+ except ValueError:
+ return None
+ if parsed.tzinfo is None:
+ parsed = parsed.replace(tzinfo=timezone.utc)
+ return parsed.timestamp()
+
+
+def make_entry(segment: str, is_dir: bool, raw: Optional[Mapping] = None, size: Optional[int] = None,
+ created: Any = None, modified: Any = None, writable: Optional[bool] = None) -> dict:
+ """Build an entry dict. ``segment`` is the entry's own path segment; ``ls`` prefixes the
+ parent path later.
+
+ ``writable`` is set only where RSpace tells us something specific (a signed ELN document
+ is locked). Left None, the key is absent, which means "unknown" rather than a guess: a
+ record shared read-only with you is equally unwritable and looks no different from here.
+ """
+ entry: Dict[str, Any] = {
+ "name": segment,
+ "type": "directory" if is_dir else "file",
+ "size": size,
+ # Galaxy and most fsspec consumers read mtime; RSpace files carry no lastModified,
+ # so created stands in for it there
+ "mtime": to_epoch(modified) if modified is not None else to_epoch(created),
+ "created": to_epoch(created),
+ "globalId": (raw or {}).get("globalId"),
+ "rspace": dict(raw) if raw else {},
+ }
+ if writable is not None:
+ entry["writable"] = writable
+ return entry
+
+
+@dataclass
+class Target:
+ """What a path addresses, as worked out by a branch's ``_resolve``.
+
+ ``kind`` is the branch's own vocabulary (``record``, ``field``, ``document``, ``file``...),
+ ``gid`` the global ID of the addressed record, ``parent_gid`` the record a field or file
+ belongs to (an Inventory record, an ELN document), ``field_gid`` the field a file was
+ reached through, and ``section`` the Inventory root section the path went through.
+ """
+ kind: str
+ gid: Optional[str] = None
+ parent_gid: Optional[str] = None
+ field_gid: Optional[str] = None
+ section: Optional[str] = None
+
+
+def client_or_new(client, factory, server: Optional[str], api_key: Optional[str], what: str):
+ """The client that was passed in, or a new one from ``server`` and ``api_key``."""
+ if client is not None:
+ return client
+ if not server or not api_key:
+ raise ValueError(f"{what} needs server and api_key, or a client")
+ return factory(server, api_key)
+
+
+def stream_pages(client: ClientBase, first_page: Mapping, key: str) -> Iterator[dict]:
+ """
+ Stream the records of a paginated RSpace response.
+
+ ``first_page`` is the first response, ``key`` the name of the collection inside it
+ ("records", "containers", "samples", ...). The next API page is fetched only when the
+ consumer reaches it, so a caller that stops after a handful of entries never pays for
+ the rest of the listing.
+ """
+ page = first_page
+ followed = set()
+ while True:
+ if key not in page:
+ # a page without its collection is malformed, not empty: _is_empty gates rmdir on this
+ raise OSError(errno.EIO, f"RSpace listing page carries no {key!r}")
+ yield from page[key]
+ # not client.link_exists(): that raises when a page carries no _links at all
+ if not any(link.get("rel") == "next" for link in page.get("_links") or []):
+ return
+ next_link = client.get_link(page, "next")
+ if next_link in followed:
+ raise OSError(errno.EIO, f"RSpace listing 'next' links form a loop: {next_link}")
+ followed.add(next_link)
+ page = client.get_link_contents(page, "next")
+
+
+# ------------------------------------------------------------------ files
+
+
+class RSpaceWriteFile(AbstractBufferedFile):
+ """A file opened for writing: everything is buffered and sent in one request when the
+ file is closed (autocommit) or when the surrounding transaction commits. RSpace takes
+ one multipart body per upload, so there is nothing to send chunk by chunk."""
+
+ def _initiate_upload(self) -> None:
+ pass
+
+ def _upload_chunk(self, final: bool = False) -> bool:
+ if final and self.autocommit:
+ self._send()
+ return False # keep the buffer; one body per upload
+
+ def commit(self) -> None:
+ self._send()
+
+ def discard(self) -> None:
+ self.buffer = io.BytesIO()
+
+ def _send(self) -> None:
+ self.buffer.seek(0)
+ self.fs.upload_fileobj(self.path, self.buffer)
+
+
+class RSpaceFSBase(AbstractFileSystem):
+ """Common behaviour for all RSpace filesystems. Subclasses implement the RSpace calls.
+
+ Paths are fsspec paths: no protocol, no leading slash, the root is ``""``.
+ """
+
+ #: Instances hold an API key: never keep them in fsspec's process-wide instance cache.
+ cachable = False
+ root_marker = ""
+ #: How many resolved name lookups to remember before starting over. Deep trees browsed
+ #: top-down reuse the same handful of entries, so a small bound is plenty.
+ NAME_CACHE_LIMIT = 4096
+ #: ``Target.kind`` values that are files: listing one raises ``NotADirectoryError``.
+ FILE_KINDS: Tuple[str, ...] = ()
+ #: Field types (lower case) that can hold a file; other fields are not browsable.
+ FILE_BEARING_FIELD_TYPES: Tuple[str, ...] = ()
+ #: False where the RSpace API cannot delete a file (the Gallery): a move is then refused
+ #: before anything is copied, since the second half could never happen.
+ CAN_DELETE_FILES = True
+ #: True where a Gallery file can be attached by reference (``link``) instead of copied.
+ CAN_LINK = False
+
+ def __init_subclass__(cls, **kwargs) -> None:
+ super().__init_subclass__(**kwargs)
+ # every subclass gets the translation on the methods it defines or overrides, so a new
+ # branch cannot forget it
+ translate_class(cls)
+
+ def __init__(self, *, writable: bool = False, allow_delete: bool = False,
+ path_style: str = "name", **storage_options: Any) -> None:
+ # fsspec caches directory listings by default and never expires them; this
+ # filesystem's own name cache is cleared on every write, so leave listings uncached
+ # unless the caller (Galaxy's cache options, say) asks for them
+ storage_options.setdefault("use_listings_cache", False)
+ super().__init__(**storage_options)
+ if path_style not in paths.PATH_STYLES:
+ raise ValueError(f"path_style must be one of {paths.PATH_STYLES}, got {path_style!r}")
+ self.writable = writable
+ self.allow_delete = allow_delete
+ #: "name" -> segments look like 'Images'; "labelled" -> 'Images [GF12]'; "id" -> 'GF12'.
+ #: See paths.py.
+ self.path_style = path_style
+ #: (parent path, segment) -> global ID, for segments written as a plain name. A name
+ #: can only be resolved against its parent's listing, so without this every operation
+ #: on a deep path would re-list every directory above it.
+ self._name_cache: Dict[Tuple[str, str], str] = {}
+
+ # ------------------------------------------------------------ paths
+
+ @classmethod
+ def _strip_protocol(cls, path):
+ if isinstance(path, list):
+ return [cls._strip_protocol(p) for p in path]
+ path = super()._strip_protocol(path)
+ return path.strip("/")
+
+ @staticmethod
+ def _join(parent: str, segment: str) -> str:
+ return f"{parent}/{segment}" if parent else segment
+
+ def _segment(self, name, gid: str) -> str:
+ """Path segment for a record, in this filesystem's path style."""
+ return paths.segment_for(self.path_style, name, gid)
+
+ def _upload_target(self, path: str) -> Tuple[str, str]:
+ """Split an upload path into the container to put the file in and the name to give
+ it: the last segment is the name of the file to create and its parent is the
+ container (a Gallery folder, an Inventory record, a document field)."""
+ name = paths.last_segment(path)
+ if not name or paths.is_root(path):
+ raise IsADirectoryError(errno.EISDIR, f"{path!r}: a file path is needed to upload to", path)
+ return self._parent(path), name
+
+ @staticmethod
+ def _upload_kwargs(name: str) -> dict:
+ """Extra keywords for the client's upload call: the name to store the file under."""
+ return {"filename": name}
+
+ # ------------------------------------------------------------ write guard
+
+ def _require_writable(self, path: str) -> None:
+ if not self.writable:
+ raise ReadOnlyError(
+ errno.EROFS, f"{path!r}: this filesystem was opened read-only; pass writable=True "
+ f"to allow writes", str(path))
+
+ def _require_delete(self, path: str) -> None:
+ if not self.allow_delete:
+ raise ReadOnlyError(
+ errno.EROFS, f"{path!r}: deletion is disabled; pass allow_delete=True to allow it", str(path))
+
+ @property
+ def read_only(self) -> bool:
+ """Read-only means nothing here can change, so a filesystem that may delete is not
+ read-only even when it refuses uploads. The two flags are deliberately separate."""
+ return not (self.writable or self.allow_delete)
+
+ # ------------------------------------------------------------ listing
+
+ def scandir(self, path: str) -> Iterator[dict]:
+ """Yield an entry per child from a single RSpace listing call, lazily: a section or
+ folder with thousands of records is fetched a page at a time as the iterator is
+ consumed. ``ls`` is this, collected into a list."""
+ path = self._strip_protocol(path)
+ return self._listing(path)
+
+ def ls(self, path: str, detail: bool = True, refresh: bool = False, **kwargs) -> list:
+ path = self._strip_protocol(path)
+ out = None if refresh else self.dircache.get(path)
+ if out is None:
+ # the directory is resolved before the list is built, so a bad path fails here
+ out = list(self._listing(path))
+ if self.dircache.use_listings_cache:
+ self.dircache[path] = out
+ return out if detail else [entry["name"] for entry in out]
+
+ def _listing(self, path: str, names_only: bool = False) -> Iterator[dict]:
+ """``_children`` with full names and the guarantee that no two children share a
+ segment and that every segment addresses the record it shows.
+
+ RSpace lets siblings share a name, so under the ``name`` path style a plain listing
+ could show the same segment twice, which would make one of the two unreachable. The
+ first record to use a name keeps it and every later one gains its global ID before
+ the extension (``data [GL112].csv``), which matches how ``_lookup`` resolves a bare
+ name. A name that itself parses as an address (``report [GL111].txt``) is given its
+ real ID the same way, in ``paths.segment_for``, so that a consumer which stores the
+ segment and resolves it later (Galaxy stores URIs) always gets the record it saw.
+ The rule needs no lookahead, so listings stay lazy.
+
+ ``names_only`` says the listing is read only to resolve a name or to test for
+ emptiness, so branches may skip per-entry extras (the Gallery's size fetches).
+ """
+ # _children resolves the directory before returning its generator, so that a bad
+ # path fails on the call rather than on first iteration; keep that by not making
+ # this a generator
+ target = self._resolve(path)
+ if target.kind in self.FILE_KINDS:
+ raise NotADirectoryError(errno.ENOTDIR, f"{path!r} is a file", path)
+ return self._named(path, self._children(target, names_only))
+
+ def _named(self, parent: str, children: Iterator[dict]) -> Iterator[dict]:
+ """Prefix each child with ``parent``, make segments unique under the ``name`` style
+ and remember every (parent, segment) -> global ID, so that one listing resolves all
+ of its children for later calls."""
+ seen = set()
+ for entry in children:
+ gid = entry.get("globalId")
+ if self.path_style == "name" and entry["name"] in seen and gid:
+ entry["name"] = paths.disambiguated_segment(entry["name"], gid)
+ seen.add(entry["name"])
+ if gid:
+ self._remember(parent, entry["name"], gid)
+ entry["name"] = self._join(parent, entry["name"])
+ yield entry
+
+ def _remember(self, parent: str, segment: str, gid: str) -> None:
+ if len(self._name_cache) >= self.NAME_CACHE_LIMIT:
+ self._name_cache.clear()
+ # both spellings, so a path stored before a document was signed still resolves
+ self._name_cache[(parent, segment)] = gid
+ self._name_cache[(parent, paths.unmarked(segment))] = gid
+
+ def _resolve(self, path: str) -> Target:
+ """What ``path`` addresses. Branches implement it with their own vocabulary."""
+ raise NotImplementedError(f"{type(self).__name__} does not implement _resolve")
+
+ def _children(self, target: Target, names_only: bool = False) -> Iterator[dict]:
+ """The entries inside a directory target, with segment names, from one listing call
+ where possible."""
+ raise NotImplementedError(f"{type(self).__name__} does not implement _children")
+
+ def _record(self, gid: str) -> dict:
+ """The raw RSpace record that owns fields (an Inventory record, an ELN document)."""
+ raise NotImplementedError(f"{type(self).__name__} does not implement _record")
+
+ def _file_fields(self, record: Mapping) -> List[dict]:
+ """The record's fields that can hold a file. Other field types are not browsable."""
+ return [f for f in record.get("fields") or []
+ if str(f.get("type", "")).lower() in self.FILE_BEARING_FIELD_TYPES and f.get("globalId")]
+
+ def _field(self, owner_gid: str, field_gid: str, record: Optional[Mapping] = None) -> dict:
+ record = record if record is not None else self._record(owner_gid)
+ return self._find_field(self._file_fields(record), field_gid, owner_gid)
+
+ def _gids(self, segments: List[str], start: int = 0) -> Iterator[Tuple[int, str]]:
+ """``(index, global id)`` for each segment from ``start`` on. A segment carrying a
+ global ID says what it addresses; a bare name is looked up in its parent's listing."""
+ for index in range(start, len(segments)):
+ gid, _ = paths.parse_segment(segments[index])
+ if gid is None:
+ gid = self._lookup("/".join(segments[:index]), segments[index])
+ yield index, gid
+
+ def _entry(self, record: Mapping, is_dir: bool, gid: Optional[str] = None,
+ name: Optional[str] = None, **extra: Any) -> dict:
+ """The entry for an RSpace record, named after it in this filesystem's path style."""
+ gid = gid or record["globalId"]
+ return make_entry(self._segment(record.get("name") if name is None else name, gid), is_dir,
+ raw=record, size=record.get("size"), created=record.get("created"),
+ modified=record.get("lastModified"), **extra)
+
+ @staticmethod
+ def _find_field(fields: Iterable[Mapping], field_gid: str, owner: str) -> dict:
+ for field in fields:
+ if field.get("globalId") == field_gid:
+ return field
+ raise FileNotFoundError(errno.ENOENT, f"no field {field_gid} on {owner}", f"{owner}/{field_gid}")
+
+ def _is_empty(self, path: str) -> bool:
+ """True if the directory has no children, without listing all of them."""
+ return next(self._listing(path, names_only=True), None) is None
+
+ # ------------------------------------------------------------ resolving a segment
+
+ def _lookup(self, parent: str, segment: str) -> str:
+ """Global ID of the child of ``parent`` whose path segment is ``segment``.
+
+ Only needed for a segment written as a plain name: one carrying a global ID says
+ what it addresses. Costs one listing of the parent, remembered afterwards; the
+ listing records every child it passes, so siblings resolve for free.
+ """
+ key = (parent, segment)
+ if key not in self._name_cache:
+ for _ in self._listing(parent, names_only=True):
+ if key in self._name_cache:
+ break
+ else:
+ raise FileNotFoundError(errno.ENOENT, f"{self._join(parent, segment)!r} not found",
+ self._join(parent, segment))
+ return self._name_cache[key]
+
+ def _gid_at(self, path: str) -> str:
+ """Global ID addressed by the last segment of ``path``, whichever style wrote it."""
+ segment = paths.last_segment(path)
+ gid, _ = paths.parse_segment(segment)
+ return gid if gid is not None else self._lookup(self._parent(path), segment)
+
+ def clear_name_cache(self) -> None:
+ """Forget resolved name lookups. Called after every change made through this
+ filesystem; call it yourself if records are renamed elsewhere while it is open."""
+ self._name_cache.clear()
+
+ def invalidate_cache(self, path: Optional[str] = None) -> None:
+ """Drop cached listings (fsspec's) and resolved names (ours).
+
+ fsspec's default does nothing to the listings cache, so a write through this
+ filesystem would otherwise leave a cached listing showing the old state. Writes are
+ rare next to reads, so everything is dropped rather than just the paths involved.
+ """
+ super().invalidate_cache(path)
+ self.dircache.clear()
+ self.clear_name_cache()
+
+ # ------------------------------------------------------------ info
+
+ def info(self, path: str, **kwargs) -> dict:
+ path = self._strip_protocol(path)
+ entry = self._info_of(path, self._resolve(path))
+ entry["name"] = path
+ return entry
+
+ def _info_of(self, path: str, target: Target) -> dict:
+ """The entry for a resolved path. Branches implement it."""
+ raise NotImplementedError(f"{type(self).__name__} does not implement _info_of")
+
+ def _stamp(self, path: str, key: str) -> Optional[datetime]:
+ stamp = self.info(path).get(key)
+ return datetime.fromtimestamp(stamp, tz=timezone.utc) if stamp is not None else None
+
+ def created(self, path: str) -> Optional[datetime]:
+ return self._stamp(path, "created")
+
+ def modified(self, path: str) -> Optional[datetime]:
+ return self._stamp(path, "mtime")
+
+ # ------------------------------------------------------------ search
+
+ #: ``/**`` with no other wildcard anywhere (a backslash escape inside the text
+ #: is allowed, which is how Galaxy sends a ``[``).
+ _GALAXY_SEARCH = re.compile(r"^(?P[^*?]*?)/?\*(?P[^*?/]+)\*$", re.S)
+
+ def glob(self, path: str, maxdepth: Optional[int] = None, **kwargs):
+ """fsspec glob, with one special case: the ``/**`` pattern a search box
+ sends (Galaxy builds exactly this) is answered by one listing of ```` and a
+ case-insensitive substring match on the last segment, in listing order. fsspec's
+ own glob treats ``[`` as a character class and ignores the backslash Galaxy escapes
+ it with, so a segment carrying ``[GL83]`` could never be found that way."""
+ path = self._strip_protocol(path)
+ # Galaxy wraps a query as ``**``, so a query that itself starts or ends with a
+ # star ("*.csv") arrives as ``**.csv*``, which fsspec refuses because ``**`` only means
+ # something as a whole path component. Inside a segment the user meant a single star.
+ path = "/".join(seg if seg == "**" else re.sub(r"\*{2,}", "*", seg) for seg in path.split("/"))
+ match = self._GALAXY_SEARCH.match(path)
+ if match and maxdepth in (None, 1):
+ needle = re.sub(r"\\(.)", r"\1", match.group("text")).lower()
+ found = {entry["name"]: entry for entry in self._listing(match.group("dir"))
+ if needle in paths.last_segment(entry["name"]).lower()}
+ return found if kwargs.get("detail") else list(found)
+ return super().glob(path, maxdepth=maxdepth, **kwargs)
+
+ # ------------------------------------------------------------ files
+
+ def _file_id(self, path: str) -> str:
+ """Numeric id of the file at ``path``; ``IsADirectoryError`` for anything else."""
+ target = self._resolve(path)
+ if target.kind not in self.FILE_KINDS:
+ raise IsADirectoryError(errno.EISDIR, f"{path!r} is a directory", path)
+ # A plain name was found in its parent's listing, so the parent holds it. A segment
+ # carrying a global ID is taken at its word by _resolve, and would otherwise reach
+ # any file the key can read, or a server error for one that does not exist.
+ if paths.parse_segment(paths.last_segment(path))[0] is not None:
+ self._placed_file(path, target)
+ return target.gid[2:]
+
+ def _placed_file(self, path: str, target: Target) -> dict:
+ """The record of the file ``target`` as the directory ``path`` reaches it through holds
+ it; ``FileNotFoundError`` when that directory does not hold it. Branches implement it."""
+ raise NotImplementedError(f"{type(self).__name__} does not implement _placed_file")
+
+ def _download(self, file_id: str, file: BinaryIO, chunk_size: int) -> None:
+ """The client call that streams file ``file_id`` into ``file``. Branches implement it."""
+ raise NotImplementedError(f"{type(self).__name__} does not implement _download")
+
+ def download_fileobj(self, path: str, file: BinaryIO, chunk_size: int = 128) -> None:
+ """Stream the file at ``path`` into the open binary ``file``."""
+ self._download(self._file_id(self._strip_protocol(path)), file, chunk_size)
+
+ def upload_fileobj(self, path: str, file: BinaryIO, **options: Any):
+ """Write the open binary ``file`` to the file at ``path``: the last segment is the name
+ to give it and its parent is the container to put it in. Branches implement it."""
+ raise NotImplementedError(f"{type(self).__name__} cannot upload")
+
+ def cat_file(self, path: str, start: Optional[int] = None, end: Optional[int] = None, **kwargs) -> bytes:
+ buffer = io.BytesIO()
+ self.download_fileobj(path, buffer)
+ data = buffer.getvalue()
+ return data[start:end] if (start is not None or end is not None) else data
+
+ def get_file(self, rpath: str, lpath, callback=None, outfile=None, **kwargs) -> None:
+ """Download straight into the local file, without the whole-file buffer ``open`` needs."""
+ if isinstance(lpath, (str, bytes)) or hasattr(lpath, "__fspath__"):
+ rpath = self._strip_protocol(rpath)
+ if self._resolve(rpath).kind not in self.FILE_KINDS:
+ os.makedirs(lpath, exist_ok=True)
+ return
+ with open(lpath, "wb") as handle:
+ self.download_fileobj(rpath, handle)
+ else:
+ self.download_fileobj(rpath, lpath)
+
+ def put_file(self, lpath, rpath: str, callback=None, mode: str = "overwrite", **kwargs) -> None:
+ """Upload the local file as the RSpace file at ``rpath``."""
+ if os.path.isdir(lpath):
+ # fsspec's put() hands directories through put_file as well, and relies on the
+ # backend to create them before their files arrive
+ self.makedirs(rpath, exist_ok=True)
+ return
+ with open(lpath, "rb") as handle:
+ self.upload_fileobj(rpath, handle, **kwargs)
+
+ def _open(self, path: str, mode: str = "rb", block_size=None, autocommit: bool = True,
+ cache_options=None, **kwargs):
+ """Reads buffer the whole file (RSpace serves no byte ranges). Writes buffer until the
+ file is closed, or until the transaction commits, and upload in one request."""
+ path = self._strip_protocol(path)
+ if mode in ("rb", "r"):
+ return io.BytesIO(self.cat_file(path))
+ if mode in ("wb", "xb", "w", "x"):
+ self._require_writable(path)
+ if not paths.last_segment(path):
+ raise IsADirectoryError(errno.EISDIR, f"{path!r} is a directory", path)
+ if mode.startswith("x") and self.exists(path):
+ raise FileExistsError(errno.EEXIST, f"{path!r} exists", path)
+ return RSpaceWriteFile(self, path, mode="wb", block_size=2 ** 62, autocommit=autocommit)
+ raise ValueError(f"mode {mode!r} is not supported; use 'rb' or 'wb'")
+
+ # ------------------------------------------------------------ folders (templates)
+
+ @staticmethod
+ def _new_folder(path: str) -> Tuple[str, str]:
+ """``(parent path, folder name)`` for mkdir. 'New [GF1]' creates 'New', as does
+ 'New'; a bare global ID is refused because it names nothing to create."""
+ segment = paths.last_segment(path)
+ gid, label = paths.parse_segment(segment)
+ if gid is not None and label is None:
+ raise ValueError(f"{path!r}: mkdir takes a folder name, not a global ID")
+ return RSpaceFSBase._parent(path), label if gid is not None else segment
+
+ def _create_folder(self, path: str, name: str, siblings: Iterable[Mapping],
+ create: Callable[[], Mapping], folder_prefixes: Tuple[str, ...],
+ exist_ok: bool) -> str:
+ """Create ``name`` unless a folder of that name is already among ``siblings``. RSpace
+ would happily make a second folder with the same name; the filesystem contract says
+ to refuse instead (or, with ``exist_ok``, to return the existing one). Returns the
+ path of the folder, in this filesystem's path style."""
+ parent = self._parent(path)
+ for record in siblings:
+ if record.get("name") == name and record["globalId"][:2] in folder_prefixes:
+ if exist_ok:
+ return self._join(parent, self._segment(name, record["globalId"]))
+ raise FileExistsError(errno.EEXIST, f"{path!r} exists", path)
+ created = create()
+ gid = created.get("globalId") or f"{folder_prefixes[0]}{created['id']}"
+ self.invalidate_cache()
+ return self._join(parent, self._segment(name, gid))
+
+ def _remove_folder(self, path: str, folder_gid: Optional[str], delete: Callable[[str], Any],
+ not_a_folder: Exception) -> None:
+ """Delete the empty folder at ``path``. ``folder_gid`` is its global ID, or None when
+ the path is not a removable folder, in which case ``not_a_folder`` is raised."""
+ if paths.is_root(path):
+ raise PermissionError(errno.EPERM, "the root cannot be removed", path)
+ if folder_gid is None:
+ raise not_a_folder
+ if not self._is_empty(path):
+ raise OSError(errno.ENOTEMPTY, f"{path!r} is not empty", path)
+ delete(folder_gid[2:])
+ self.invalidate_cache()
+
+ # ------------------------------------------------------------ mutations (defaults)
+
+ def mkdir(self, path: str, create_parents: bool = True, **kwargs) -> None:
+ raise NotImplementedError(f"{type(self).__name__} cannot create directories")
+
+ def makedirs(self, path: str, exist_ok: bool = False) -> None:
+ """Only the last segment is created; RSpace folders are not made in bulk here."""
+ path = self._strip_protocol(path)
+ if exist_ok and self.exists(path):
+ if not self.isdir(path):
+ raise FileExistsError(errno.EEXIST, f"{path!r} exists and is not a directory", path)
+ return
+ self.mkdir(path, create_parents=False, exist_ok=exist_ok)
+
+ def rmdir(self, path: str) -> None:
+ raise NotImplementedError(f"{type(self).__name__} cannot delete directories")
+
+ def rm_file(self, path: str) -> None:
+ raise NotImplementedError(f"{type(self).__name__} cannot delete files")
+
+ def rm(self, path, recursive: bool = False, maxdepth: Optional[int] = None) -> None:
+ """Remove files and empty directories. Recursive removal is refused: RSpace records
+ are not something a filesystem should delete wholesale."""
+ if recursive:
+ raise NotImplementedError("recursive removal is not supported on RSpace")
+ for one in ([path] if isinstance(path, str) else path):
+ one = self._strip_protocol(one)
+ if self._resolve(one).kind in self.FILE_KINDS:
+ self.rm_file(one)
+ else:
+ self.rmdir(one)
+
+ def link(self, path: str, media_file_gid: str) -> dict:
+ """Attach a Gallery file by reference. Only the branches with ``CAN_LINK`` can."""
+ raise NotImplementedError(f"{path!r}: the Gallery holds the files, it cannot link to them")
+
+ def cp_file(self, path1: str, path2: str, **kwargs) -> None:
+ """Copy a file within this filesystem by downloading and uploading it."""
+ path1, path2 = self._strip_protocol(path1), self._strip_protocol(path2)
+ self._require_writable(path2)
+ self.upload_fileobj(path2, io.BytesIO(self.cat_file(path1)))
+
+ def _require_movable(self, path1: str, recursive: bool) -> None:
+ """A move is a copy followed by a remove, so check that the remove is allowed
+ *before* uploading: otherwise the copy succeeds and the caller is left with an
+ error and a duplicate the API may not be able to delete."""
+ self._require_delete(path1)
+ if not self.CAN_DELETE_FILES:
+ raise NotImplementedError(
+ f"{path1!r} cannot be moved: the RSpace API cannot delete a Gallery file, so only "
+ f"a copy is possible")
+ if recursive:
+ raise NotImplementedError("recursive move is not supported on RSpace")
+
+ def mv(self, path1: str, path2: str, recursive: bool = False, maxdepth: Optional[int] = None,
+ **kwargs) -> None:
+ self._require_writable(path2)
+ self._require_movable(path1, recursive)
+ self.cp_file(path1, path2)
+ self.rm_file(path1)
+
+
+# ``__init_subclass__`` only sees subclasses; the methods the base defines itself (``ls``,
+# ``info``, ``cat_file``...) are translated here, once, so a branch that inherits them
+# unchanged still raises ``OSError`` rather than a client exception.
+translate_class(RSpaceFSBase)
diff --git a/rspace_client/fs/gallery.py b/rspace_client/fs/gallery.py
new file mode 100644
index 0000000..3315fb1
--- /dev/null
+++ b/rspace_client/fs/gallery.py
@@ -0,0 +1,405 @@
+"""
+fsspec view of the RSpace Gallery.
+
+Gallery folders (``GF``) are directories and Gallery files (``GL``) are files. The
+filesystem root is the Gallery's top folder, resolved lazily on first use, by
+name, from the user's Home folder listing.
+
+ fs = GalleryFilesystem(server, api_key) # read-only
+ fs = GalleryFilesystem(server, api_key, writable=True) # upload / mkdir allowed
+ fs = GalleryFilesystem(eln_client=my_client) # share a client (unified mount)
+ fs = GalleryFilesystem(server, api_key, path_style="id") # bare 'GF12' segments instead of 'Images'
+
+Listings are one request per directory. The folder-tree endpoint omits file sizes, and
+consumers such as Galaxy need an integer size for every file, so by default each file in a
+listing is topped up with its own record (``fetch_sizes=True``); pass ``fetch_sizes=False``
+for a listing that stays at one request and reports ``size`` as None.
+
+Gallery upload routing: the Gallery is split into media-type sections
+(Images, Documents, Chemistry, ...) and a file may only be placed in a folder of the
+matching section. ``upload_fileobj()`` returns a :class:`Placement`; when the target
+folder's section rejects the file it either raises :class:`GallerySectionMismatch`
+(``on_mismatch="raise"``, the default) or lets RSpace place the file in the correct
+section and reports where it landed (``on_mismatch="reroute"``).
+"""
+from __future__ import annotations
+
+import errno
+import logging
+from dataclasses import dataclass
+from typing import Any, BinaryIO, Iterator, Mapping, Optional
+
+from rspace_client.client_base import ClientBase
+from rspace_client.eln import eln
+from rspace_client.exceptions import RSpaceError
+
+from . import paths
+from .base import RSpaceFSBase, Target, client_or_new, deletes, make_entry, stream_pages, writes
+
+logger = logging.getLogger(__name__)
+
+FOLDER_PREFIXES = ("GF", "FL", "NB")
+FILE_PREFIX = "GL"
+#: The Gallery root is the folder of this name in the user's Home folder listing.
+GALLERY_FOLDER_NAME = "Gallery"
+
+# Accepted values for the ``on_mismatch`` policy.
+ON_MISMATCH_RAISE = "raise"
+ON_MISMATCH_REROUTE = "reroute"
+_ON_MISMATCH_VALUES = (ON_MISMATCH_RAISE, ON_MISMATCH_REROUTE)
+
+# The RSpace Gallery's media-type sections. "Miscellaneous" is the catch-all
+# that accepts any file not matching a more specific section; the others accept
+# only their listed extensions (including "Documents", which is a fixed set, not
+# a catch-all).
+MISCELLANEOUS_SECTION = "Miscellaneous"
+
+# Best-effort mapping of file extension to the Gallery section RSpace classifies
+# it into, taken from the Gallery documentation. This mirrors the server's own
+# classification but is NOT authoritative: it is used only to phrase error
+# messages, never to decide whether an upload is allowed. Any extension not
+# listed falls through to "Miscellaneous", matching the server default.
+_SECTION_BY_EXTENSION = {
+ # Images
+ "png": "Images", "jpg": "Images", "jpeg": "Images", "gif": "Images",
+ "bmp": "Images", "tif": "Images", "tiff": "Images",
+ # Audios
+ "mp3": "Audios", "wav": "Audios", "wma": "Audios", "aac": "Audios",
+ "ogg": "Audios",
+ # Videos
+ "mp4": "Videos", "mov": "Videos", "hdmov": "Videos", "m4v": "Videos",
+ "wmv": "Videos", "avi": "Videos", "mpg": "Videos", "mpeg": "Videos",
+ "flv": "Videos", "3gp": "Videos",
+ # Documents (a fixed set, NOT a catch-all)
+ "doc": "Documents", "docx": "Documents", "rtf": "Documents",
+ "pdf": "Documents", "odt": "Documents", "ods": "Documents",
+ "odp": "Documents", "txt": "Documents", "ppt": "Documents",
+ "pptx": "Documents", "xls": "Documents", "xlsx": "Documents",
+ "csv": "Documents", "pps": "Documents", "md": "Documents",
+ # Chemistry (documented subset; the server accepts more)
+ "skc": "Chemistry", "mrv": "Chemistry", "cxsmiles": "Chemistry",
+ "cxsmarts": "Chemistry", "cdx": "Chemistry", "cdxml": "Chemistry",
+ "csrdf": "Chemistry", "cml": "Chemistry",
+}
+
+
+def classify_media_section(filename: Optional[str]) -> Optional[str]:
+ """
+ Guess which Gallery section RSpace will file ``filename`` under, mirroring the
+ server-side classification. Returns a section name (e.g. "Images"),
+ "Miscellaneous" as the catch-all when the extension is not one of a specific
+ section's types, or None when no filename/extension is available to guess
+ from.
+
+ The server remains the authority on placement; this is only used to make
+ error messages and log lines more helpful.
+ """
+ if not filename or "." not in filename:
+ return None
+ ext = filename.rsplit(".", 1)[-1].lower()
+ return _SECTION_BY_EXTENSION.get(ext, MISCELLANEOUS_SECTION)
+
+
+def _a_or_an(word: str) -> str:
+ return "an" if word[:1].upper() in "AEIOU" else "a"
+
+
+def check_policy(on_mismatch: str) -> str:
+ if on_mismatch not in _ON_MISMATCH_VALUES:
+ raise ValueError("on_mismatch must be one of {}".format(_ON_MISMATCH_VALUES))
+ return on_mismatch
+
+
+class GallerySectionMismatch(ClientBase.ApiError):
+ """
+ Raised when a file could not be uploaded into the requested Gallery folder
+ because that folder belongs to a media-type section that does not accept the
+ file. Carries the folder's section and, when it could be guessed, the file's
+ media type, so callers can react programmatically as well as read the message.
+
+ It is deliberately *not* translated into an ``OSError``: the message is written for the
+ person in front of the screen and the attributes are what a library caller needs.
+ """
+
+ fs_passthrough = True
+
+ def __init__(self, message, *, folder_section=None, folder_global_id=None,
+ file_media_type=None, response_status_code=None):
+ super().__init__(message, response_status_code=response_status_code)
+ self.folder_section = folder_section
+ self.folder_global_id = folder_global_id
+ self.file_media_type = file_media_type
+
+
+@dataclass
+class Placement:
+ """Where an uploaded file actually ended up in the Gallery.
+
+ Returned by :meth:`GalleryFilesystem.upload_fileobj`. When ``rerouted`` is True the
+ file could not go into the requested folder and RSpace filed it in the section its
+ media type belongs to; ``path`` is a readable description of where that is.
+ """
+ file_global_id: Optional[str]
+ folder_global_id: Optional[str]
+ section: Optional[str]
+ path: str
+ rerouted: bool
+ requested_path: Optional[str] = None
+
+
+def folder_section(eln_client: eln.ELNClient, folder_id) -> Optional[str]:
+ """The Gallery section (mediaType) a folder belongs to, or None if it cannot be
+ determined. Best-effort: never raises, so it cannot mask the real outcome of an upload."""
+ try:
+ return eln_client.get_folder(folder_id).get("mediaType")
+ except RSpaceError:
+ return None
+
+
+def human_path(folder: Mapping[str, Any]) -> str:
+ """Best-effort readable path for a folder, e.g. 'Gallery/Documents/Api Inbox'. Uses the
+ API's pathToRootFolder when present, otherwise the section and folder name."""
+ trail = folder.get("pathToRootFolder")
+ names = [f.get("name") for f in trail if f.get("name")] if isinstance(trail, list) else []
+ if names:
+ return "/".join(names)
+ return "/".join(["Gallery"] + [folder[k] for k in ("mediaType", "name") if folder.get(k)])
+
+
+def placement(eln_client: eln.ELNClient, response: Any, requested_path: Optional[str],
+ rerouted: bool) -> Placement:
+ """Build a Placement from an upload response, resolving the parent folder for
+ section/path feedback where possible. Tolerates a response that is not a dict (e.g.
+ None): a caller may wrap ``upload_file`` and discard its return value, and Placement
+ construction must never crash a successful upload."""
+ if not isinstance(response, dict):
+ response = {}
+ folder: Mapping[str, Any] = {}
+ parent_id = response.get("parentFolderId")
+ if parent_id is not None:
+ try:
+ folder = eln_client.get_folder(parent_id)
+ except RSpaceError:
+ pass
+ return Placement(
+ file_global_id=response.get("globalId"),
+ folder_global_id=folder.get("globalId"),
+ section=folder.get("mediaType"),
+ path=human_path(folder) if folder else "Gallery",
+ rerouted=rerouted,
+ requested_path=requested_path,
+ )
+
+
+def mismatch_message(named: str, container: str, section: str, guessed: Optional[str], err: Exception) -> str:
+ """The explanation a person reads when a folder's section refuses their file."""
+ message = (f"Could not upload {named} to Gallery folder {container or '/'}. That folder "
+ f"is in the '{section}' section, which only accepts {section} files")
+ if guessed and guessed != section:
+ looks = ("does not match a specialised section and belongs in 'Miscellaneous'"
+ if guessed == MISCELLANEOUS_SECTION else f"looks like {_a_or_an(guessed)} '{guessed}' file")
+ message += f", but {named} {looks}"
+ remedy = f"Choose a folder in the '{guessed}' section instead"
+ else:
+ remedy = "Choose a folder in the section that matches this file type"
+ # Lead with what the person in front of the screen can act on; the API-level option is
+ # a parenthetical for library callers, who are not always the audience (this message
+ # reaches Galaxy users verbatim).
+ return message + (f". {remedy}, or upload without choosing a folder and RSpace will file it in the "
+ f"right section itself (library callers can pass on_mismatch='reroute' to do that "
+ f"automatically). Original API error: {err}")
+
+
+class GalleryFilesystem(RSpaceFSBase):
+ """Target kinds: root | folder | file."""
+
+ protocol = ("rspace-gallery",)
+ FILE_KINDS = ("file",)
+ CAN_DELETE_FILES = False # the RSpace API has no call to delete a Gallery file
+
+ def __init__(self, server: Optional[str] = None, api_key: Optional[str] = None, *,
+ eln_client: Optional[eln.ELNClient] = None, writable: bool = False,
+ allow_delete: bool = False, path_style: str = "name",
+ on_mismatch: str = ON_MISMATCH_RAISE, page_size: int = 100,
+ fetch_sizes: bool = True, **storage_options: Any) -> None:
+ super().__init__(writable=writable, allow_delete=allow_delete, path_style=path_style,
+ **storage_options)
+ #: records fetched per API request while a listing is consumed, as in the other branches
+ self.page_size = max(1, int(page_size))
+ self.eln_client = client_or_new(eln_client, eln.ELNClient, server, api_key, "GalleryFilesystem")
+ self.on_mismatch = check_policy(on_mismatch)
+ #: fetch each file's own record while listing, because the folder-tree endpoint
+ #: carries no size (one extra request per file)
+ self.fetch_sizes = fetch_sizes
+ self._gallery_id: Optional[int] = None
+
+ # ------------------------------------------------------------ RSpace plumbing
+
+ @property
+ def gallery_id(self) -> int:
+ """Numeric id of the Gallery root folder, looked up once on first use."""
+ if self._gallery_id is None:
+ # the Gallery root is a system folder with a GF id (the workspace branch hides it
+ # by the same rule); a user's own folder named "Gallery" is an FL and is skipped
+ for record in self._all_records(None):
+ if (record.get("name") == GALLERY_FOLDER_NAME
+ and str(record.get("globalId", "")).startswith("GF")):
+ self._gallery_id = int(record["id"])
+ break
+ else:
+ raise FileNotFoundError(
+ errno.ENOENT, f"no folder named {GALLERY_FOLDER_NAME!r} in the Home folder listing", "")
+ return self._gallery_id
+
+ def _all_records(self, folder_id) -> Iterator[dict]:
+ """Stream a folder's contents, following the tree listing's 'next' links only as
+ far as the consumer reads."""
+ first_page = self.eln_client.list_folder_tree(folder_id, page_size=self.page_size)
+ return stream_pages(self.eln_client, first_page, "records")
+
+ def _resolve(self, path: str) -> Target:
+ if paths.is_root(path):
+ return Target("root")
+ gid = self._gid_at(path)
+ if gid[:2] in FOLDER_PREFIXES:
+ return Target("folder", gid=gid)
+ if gid[:2] == FILE_PREFIX:
+ return Target("file", gid=gid)
+ raise FileNotFoundError(errno.ENOENT, f"{path!r} not found", path)
+
+ def _folder_id(self, path: str):
+ """Numeric folder id for a directory path (root -> Gallery root)."""
+ target = self._resolve(path)
+ if target.kind == "root":
+ return self.gallery_id
+ if target.kind != "folder":
+ raise NotADirectoryError(errno.ENOTDIR, f"{path!r} is a file", path)
+ return target.gid[2:]
+
+ def _record_entry(self, record: dict) -> dict:
+ return self._entry(record, record["globalId"][:2] in FOLDER_PREFIXES)
+
+ # ------------------------------------------------------------ listing and info
+
+ def _children(self, target: Target, names_only: bool = False) -> Iterator[dict]:
+ folder_id = self.gallery_id if target.kind == "root" else target.gid[2:]
+ return self._scan_records(folder_id, self.fetch_sizes and not names_only)
+
+ def _scan_records(self, folder_id, want_sizes: bool) -> Iterator[dict]:
+ # One listing call per directory, plus one record fetch per file when sizes are
+ # wanted: folder-tree items carry no size and consumers such as Galaxy require an
+ # integer size for every file.
+ for record in self._all_records(folder_id):
+ if want_sizes and record.get("globalId", "")[:2] == FILE_PREFIX and record.get("size") is None:
+ try:
+ record = self.eln_client.get_file_info(record["id"])
+ except ClientBase.ApiError: # keep listing even if one file cannot be read
+ logger.debug("could not read file details for %s", record.get("globalId"))
+ yield self._record_entry(record)
+
+ def _info_of(self, path: str, target: Target) -> dict:
+ if target.kind == "root":
+ return make_entry("", True, raw={"name": GALLERY_FOLDER_NAME})
+ if target.kind == "folder":
+ return self._entry(self.eln_client.get_folder(target.gid[2:]), True, target.gid)
+ return self._entry(self._placed_file(path, target), False, target.gid)
+
+ def _placed_file(self, path: str, target: Target) -> dict:
+ """The file's record, provided it sits in the folder the path names."""
+ record = self.eln_client.get_file_info(target.gid[2:])
+ if str(record.get("parentFolderId")) != str(self._folder_id(self._parent(path))):
+ raise FileNotFoundError(errno.ENOENT, f"{path!r}: {target.gid} is not in this folder", path)
+ return record
+
+ # ------------------------------------------------------------ folders
+
+ @writes
+ def mkdir(self, path: str, create_parents: bool = True, exist_ok: bool = False, **kwargs) -> None:
+ path = self._strip_protocol(path)
+ if paths.is_root(path):
+ raise FileExistsError(errno.EEXIST, "the root exists", path)
+ parent, name = self._new_folder(path)
+ folder_id = self._folder_id(parent)
+ self._create_folder(path, name, self._all_records(folder_id),
+ lambda: self.eln_client.create_folder(name, folder_id),
+ FOLDER_PREFIXES, exist_ok)
+
+ @deletes
+ def rmdir(self, path: str) -> None:
+ path = self._strip_protocol(path)
+ target = self._resolve(path)
+ self._remove_folder(path, target.gid if target.kind == "folder" else None,
+ self.eln_client.delete_folder,
+ NotADirectoryError(errno.ENOTDIR, f"{path!r} is not a folder", path))
+
+ def rm_file(self, path: str) -> None:
+ raise NotImplementedError("deleting Gallery files is not supported by the RSpace API")
+
+ # ------------------------------------------------------------ files
+
+ def _download(self, file_id: str, file: BinaryIO, chunk_size: int) -> None:
+ self.eln_client.download_file(file_id, file, chunk_size)
+
+ # ------------------------------------------------------------ upload with section routing
+
+ @writes
+ def upload_fileobj(self, path: str, file: BinaryIO, on_mismatch: Optional[str] = None,
+ **options: Any) -> Placement:
+ """
+ Write ``file`` to the Gallery file at ``path``: the last segment is the name to give
+ the file and its parent is the folder to put it in. A file directly under the root
+ is left to the API's default target, the relevant section's "Api Inbox".
+
+ :param on_mismatch: optional override of the filesystem-wide policy for
+ this call ("raise" or "reroute"); defaults to the value
+ passed to the constructor.
+ :return: a :class:`Placement` describing where the file ended up.
+
+ The RSpace Gallery is split into media-type sections (Images, Documents,
+ Chemistry, ...) and a file may only be placed in a folder whose section
+ matches the file's media type. If ``path`` names a folder in the wrong
+ section the upload is rejected. Depending on the effective policy this
+ either raises a :class:`GallerySectionMismatch` naming the folder's
+ section (``"raise"``) or places the file in the correct section's inbox
+ and returns a Placement with ``rerouted=True`` (``"reroute"``).
+ """
+ policy = check_policy(self.on_mismatch if on_mismatch is None else on_mismatch)
+ path = self._strip_protocol(path)
+ container, name = self._upload_target(path)
+ folder_id = None if paths.is_root(container) else self._folder_id(container)
+ try:
+ response = self.eln_client.upload_file(file, folder_id, **self._upload_kwargs(name))
+ self.invalidate_cache()
+ return placement(self.eln_client, response, requested_path=container, rerouted=False)
+ except ClientBase.ApiError as err:
+ section = None if folder_id is None else folder_section(self.eln_client, folder_id)
+ if section is None:
+ raise
+ filename = options.get("filename") or name
+ guessed = classify_media_section(filename)
+
+ if policy == ON_MISMATCH_REROUTE:
+ try:
+ file.seek(0)
+ except (AttributeError, OSError, ValueError):
+ pass
+ response = self.eln_client.upload_file(file, None, **self._upload_kwargs(name))
+ self.invalidate_cache()
+ placed = placement(self.eln_client, response, requested_path=container, rerouted=True)
+ logger.info(
+ "RSpace Gallery: %s could not go in %s (section '%s'); placed in %s instead",
+ f"'{filename}'" if filename else "file", container, section, placed.path)
+ return placed
+
+ # `from None` rather than `from err`: this exception reaches people through
+ # Galaxy, which prints the whole chain and clips it from the top, hiding the
+ # explanation at the bottom. The server's own reason is already quoted in the
+ # message, and the original error stays available on the exception's
+ # attributes, so nothing is lost by not stacking three tracebacks.
+ raise GallerySectionMismatch(
+ mismatch_message(f"'{filename}'" if filename else "the file", container, section, guessed, err),
+ folder_section=section,
+ folder_global_id="GF" + str(folder_id),
+ file_media_type=guessed,
+ response_status_code=getattr(err, "response_status_code", None),
+ ) from None
diff --git a/rspace_client/fs/inventory.py b/rspace_client/fs/inventory.py
new file mode 100644
index 0000000..59e5e13
--- /dev/null
+++ b/rspace_client/fs/inventory.py
@@ -0,0 +1,335 @@
+"""
+fsspec view of RSpace Inventory: every record is a folder that may hold files.
+
+ (root) benches, plus the sections Containers, Samples, Templates
+ Containers top-level containers, streamed however many there are
+ Samples, Templates likewise
+ WB user1a a bench: its containers and subsamples, plus attachments
+ .../Freezer -80 a container: child containers and subsamples as folders,
+ attachments as files
+ .../Plasmid pUC19 a sample: its subsamples as folders, attachments as files
+ .../Plasmid pUC19 #1 a subsample: attachments, plus a shortcut folder
+ .../sample: Plasmid pUC19 the shortcut: that sample's attachments only (no subsamples,
+ so the tree has no cycles)
+ .../Plasmid a sample template: attachments
+ .../microscope an instrument: attachments
+ .../Safety Data an attachment-type template field: the one file it holds
+
+Segments are the records' own names under the default path style; a segment written as a
+global ID (``IC200``, ``Freezer -80 [IC200]``) resolves whatever style is in force.
+
+Records carry files in two places: their own ``attachments``, listed as files in the record
+folder, and template-defined **attachment fields**, listed as folders holding at most one
+file each (the same shape as an ELN document field). Only ``attachment`` fields are shown;
+the other field types cannot hold a file. An attachment field accepts a file only while it
+is empty: RSpace would replace the current file rather than add to it, which a folder does
+not suggest, so uploading into an occupied field raises ``FileExistsError``. Remove the
+file first (with ``allow_delete=True``) and then upload.
+
+Samples are not stored *in* containers in RSpace, their subsamples are; the shortcut
+folder is how you get from a subsample you found in a container to its sample's files.
+
+A record listing is one API call. A section streams: its records are fetched
+``page_size`` at a time as the listing is consumed (``scandir`` keeps that laziness, ``ls``
+collects the list). Read-only by default: ``writable=True`` allows attaching files to any
+record, ``allow_delete=True`` allows removing attachments.
+"""
+from __future__ import annotations
+
+import errno
+from typing import Any, BinaryIO, Iterator, Optional
+
+from rspace_client.client_base import Pagination
+from rspace_client.eln import eln
+from rspace_client.inv import inv
+
+from . import paths
+from .base import RSpaceFSBase, Target, client_or_new, deletes, make_entry, stream_pages, writes
+
+ATTACHMENT_PREFIX = "IF"
+FIELD_PREFIX = "SF" # a template-defined field on a sample, template or instrument
+CONTAINER_PREFIXES = ("IC", "BE")
+RECORD_GETTERS = {
+ "IC": "get_container_by_id",
+ "BE": "get_workbench_by_id", # /containers/{id} answers 422 for a bench
+ "SA": "get_sample_by_id",
+ "SS": "get_subsample_by_id",
+ "IT": "get_sample_template_by_id",
+ "IN": "get_instrument_by_id", # instruments can sit on a bench and carry attachments
+}
+#: Record types the server will actually attach a file to, confirmed live against RSpace
+#: 2.27. A bench answers "Unsupported global id type" and a sample template answers "Sample
+#: Templates don't support file attachments yet". Both present as ordinary record folders, so
+#: without this the Gallery upload happens first and its file is stranded when the link is
+#: refused, and the RSpace API cannot delete a Gallery file.
+ATTACHABLE_PREFIXES = ("IC", "SA", "SS", "IN", "SF")
+SECTION_LISTERS = { # section -> (client method, collection key in the response)
+ "Containers": ("list_top_level_containers", "containers"),
+ "Samples": ("list_samples", "samples"),
+ "Templates": ("list_sample_templates", "templates"),
+}
+#: The named sections at this filesystem's root, in the order they are listed.
+SECTIONS = tuple(SECTION_LISTERS)
+
+
+class InventoryFilesystem(RSpaceFSBase):
+ """Target kinds: root | section | record | shortcut | field | attachment."""
+
+ protocol = ("rspace-inventory",)
+ FILE_KINDS = ("attachment",)
+ #: Inventory field types are number/date/string/text/uri/reference/attachment/time/radio/
+ #: choice; only "attachment" can hold a file, and it holds at most one.
+ FILE_BEARING_FIELD_TYPES = ("attachment",)
+ CAN_LINK = True
+
+ def __init__(self, server: Optional[str] = None, api_key: Optional[str] = None, *,
+ inv_client: Optional[inv.InventoryClient] = None,
+ eln_client: Optional[eln.ELNClient] = None, writable: bool = False,
+ allow_delete: bool = False, path_style: str = "name",
+ page_size: int = 100, via_gallery: Optional[bool] = None,
+ **storage_options: Any) -> None:
+ super().__init__(writable=writable, allow_delete=allow_delete, path_style=path_style,
+ **storage_options)
+ self.inv_client = client_or_new(inv_client, inv.InventoryClient, server, api_key, "InventoryFilesystem")
+ #: Put uploaded bytes in the Gallery and link them, rather than creating a file that
+ #: exists only inside Inventory. See ``upload_fileobj``. Needs an ELN client as well
+ #: as an Inventory one; left as None it turns itself on whenever one can be had, and
+ #: asking for it explicitly when one cannot is an error rather than a silent downgrade.
+ if eln_client is None and via_gallery is not False and server and api_key:
+ eln_client = eln.ELNClient(server, api_key)
+ self.eln_client = eln_client
+ if via_gallery and eln_client is None:
+ raise ValueError(
+ "via_gallery=True needs an ELN client as well as an Inventory one: pass "
+ "eln_client=, or server and api_key, or via_gallery=False to create files "
+ "that exist only inside Inventory")
+ self.via_gallery = bool(eln_client) if via_gallery is None else via_gallery
+ #: how many records to fetch per API request while streaming a listing; a
+ #: performance knob, not something a user of the filesystem sees.
+ self.page_size = max(1, int(page_size))
+
+ # ------------------------------------------------------------ path grammar
+
+ def _resolve(self, path: str) -> Target:
+ if paths.is_root(path):
+ return Target("root")
+ segments = paths.segments(path)
+ section: Optional[str] = None
+ previous_gid: Optional[str] = None
+ target: Optional[Target] = None
+ first = 0
+ if segments[0] in SECTIONS:
+ section = segments[0]
+ first = 1
+ if first == len(segments):
+ return Target("section", section=section)
+ for index, gid in self._gids(segments, first):
+ prefix = gid[:2]
+ if prefix == ATTACHMENT_PREFIX:
+ if index != len(segments) - 1: # nothing lives below a file
+ raise FileNotFoundError(errno.ENOENT, f"{path!r} not found", path)
+ return Target("attachment", gid=gid, parent_gid=previous_gid)
+ if prefix == FIELD_PREFIX:
+ if previous_gid is None:
+ raise FileNotFoundError(errno.ENOENT, f"{path!r} not found", path)
+ target = Target("field", gid=gid, parent_gid=previous_gid)
+ previous_gid = gid
+ continue
+ if prefix not in RECORD_GETTERS:
+ raise FileNotFoundError(errno.ENOENT, f"{path!r} not found", path)
+ shortcut = prefix == "SA" and previous_gid is not None and previous_gid.startswith("SS")
+ previous_gid = gid
+ target = Target("shortcut" if shortcut else "record", gid=gid, section=section)
+ if target is None:
+ raise FileNotFoundError(errno.ENOENT, f"{path!r} not found", path)
+ return target
+
+ # ------------------------------------------------------------ RSpace plumbing
+
+ def _fetch(self, gid: str, include_content: bool = True) -> dict:
+ """Fetch a record. A container's contents cost extra to assemble server-side, so
+ ``include_content=False`` when only the record itself is wanted (``info``)."""
+ getter = getattr(self.inv_client, RECORD_GETTERS[gid[:2]])
+ if gid[:2] in CONTAINER_PREFIXES and include_content:
+ return getter(gid[2:], include_content=True)
+ return getter(gid[2:])
+
+ def _record(self, gid: str) -> dict:
+ return self._fetch(gid, include_content=False)
+
+ def _section_items(self, section: str) -> Iterator[dict]:
+ """Stream a section's records, fetching the next API page only when it is reached."""
+ method, key = SECTION_LISTERS[section]
+ first_page = getattr(self.inv_client, method)(Pagination(page_number=0, page_size=self.page_size))
+ return stream_pages(self.inv_client, first_page, key)
+
+ # ------------------------------------------------------------ entry builders
+
+ def _file_entry(self, attachment: dict, gid: Optional[str] = None) -> dict:
+ return self._entry(attachment, False, gid)
+
+ @staticmethod
+ def _virtual_dir(name: str, extra: Optional[dict] = None) -> dict:
+ raw = {"name": name, "virtual": True}
+ raw.update(extra or {})
+ return make_entry(name, True, raw=raw)
+
+ def _field_entry(self, field: dict) -> dict:
+ raw = dict(field, fieldType=field.get("type"))
+ raw.pop("attachment", None)
+ return self._entry(raw, True)
+
+ def _shortcut_entry(self, sample: dict) -> dict:
+ return self._entry(dict(sample, shortcut="sample"), True, name=f"sample: {sample.get('name', '')}")
+
+ # ------------------------------------------------------------ children of a directory
+
+ def _children(self, target: Target, names_only: bool = False) -> Iterator[dict]:
+ if target.kind == "root":
+ return self._root_children()
+ if target.kind == "section":
+ return (self._entry(item, True) for item in self._section_items(target.section))
+ if target.kind == "field":
+ attachment = self._field(target.parent_gid, target.gid).get("attachment")
+ return iter([self._file_entry(attachment)] if attachment else [])
+ return self._record_children(target, self._fetch(target.gid))
+
+ def _root_children(self) -> Iterator[dict]:
+ for bench in self.inv_client.get_workbenches():
+ yield self._entry(bench, True)
+ for section in SECTIONS:
+ yield self._virtual_dir(section, {"section": section})
+
+ def _record_children(self, target: Target, record: dict) -> Iterator[dict]:
+ prefix = target.gid[:2]
+ if target.kind == "record":
+ if prefix in CONTAINER_PREFIXES:
+ for location in record.get("locations", []):
+ content = location.get("content")
+ if content and content.get("globalId"):
+ yield self._entry(content, True)
+ elif prefix == "SA":
+ for sub in record.get("subSamples", []):
+ yield self._entry(sub, True)
+ elif prefix == "SS":
+ sample = record.get("sample")
+ if sample and sample.get("globalId"):
+ yield self._shortcut_entry(sample)
+ for field in self._file_fields(record):
+ yield self._field_entry(field)
+ for attachment in record.get("attachments", []):
+ yield self._file_entry(attachment)
+
+ # ------------------------------------------------------------ info
+
+ def _info_of(self, path: str, target: Target) -> dict:
+ if target.kind == "field":
+ return self._field_entry(self._field(target.parent_gid, target.gid))
+ if target.kind == "root":
+ return make_entry("", True, raw={"name": "Inventory"})
+ if target.kind == "section":
+ return self._virtual_dir(target.section, {"section": target.section})
+ if target.kind == "attachment":
+ return self._file_entry(self._placed_file(path, target), target.gid)
+ record = self._fetch(target.gid, include_content=False)
+ if target.kind == "shortcut":
+ return self._shortcut_entry(record)
+ return self._entry(record, True, target.gid)
+
+ # ------------------------------------------------------------ files
+
+ def _placed_file(self, path: str, target: Target) -> dict:
+ """The attachment, provided it hangs off the record or field it was reached through
+ (for a shortcut, the sample). A replaced field attachment is kept by the server as
+ deleted, and is not there either."""
+ attachment = self.inv_client.get_attachment_by_id(target.gid[2:])
+ if attachment.get("parentGlobalId") != target.parent_gid or attachment.get("deleted"):
+ raise FileNotFoundError(
+ errno.ENOENT, f"{path!r}: {target.gid} is not attached to {target.parent_gid or 'this folder'}", path)
+ return attachment
+
+ def _require_empty_field(self, target: Target, container: str) -> None:
+ """An attachment field holds one file, and RSpace *replaces* it when another is added,
+ whether uploaded or linked. A folder looks like something you add to, so silently
+ destroying its contents would be a nasty surprise: refuse, and say what to do. Called
+ before anything reaches the server, so a refusal leaves nothing behind."""
+ existing = self._field(target.parent_gid, target.gid).get("attachment")
+ if existing:
+ raise FileExistsError(
+ errno.EEXIST,
+ f"{container!r} already holds {existing.get('name')!r}; an attachment field "
+ f"takes a single file and adding another would replace it. Remove the existing "
+ f"file first.", container)
+
+ @deletes
+ def rm_file(self, path: str) -> None:
+ path = self._strip_protocol(path)
+ target = self._resolve(path)
+ if target.kind != "attachment":
+ raise IsADirectoryError(errno.EISDIR, f"{path!r} is not an attachment", path)
+ self.inv_client.delete_attachment_by_id(target.gid[2:])
+ self.invalidate_cache()
+
+ def _download(self, file_id: str, file: BinaryIO, chunk_size: int) -> None:
+ self.inv_client.download_attachment_by_id(file_id, file, chunk_size)
+
+ @writes
+ def upload_fileobj(self, path: str, file: BinaryIO, **options: Any) -> dict:
+ """
+ Write ``file`` to ``path``: the last segment names the attachment to create and its
+ parent is the record (container, sample, subsample, instrument) or the attachment
+ field to put it on. Returns the new attachment record.
+
+ By default the bytes go to the **Gallery** and the record is given a link to them,
+ which is what the web interface's "link from Gallery" does and what an ELN upload
+ already did. The alternative, which the API also offers, creates a file that exists
+ only inside Inventory: invisible in the Gallery, not reusable anywhere else, and
+ impossible to find again except through the record it hangs off. Pass
+ ``via_gallery=False`` for that behaviour.
+ """
+ path = self._strip_protocol(path)
+ container, name = self._upload_target(path)
+ target = self._resolve(container)
+ if target.kind == "field":
+ self._require_empty_field(target, container)
+ elif target.kind not in ("record", "shortcut"):
+ raise NotImplementedError(
+ f"{container!r}: files attach to a record or one of its attachment fields")
+ if target.gid[:2] not in ATTACHABLE_PREFIXES:
+ # Checked before the Gallery upload: the server would refuse the link afterwards
+ # and the uploaded file could never be deleted.
+ raise NotImplementedError(
+ f"{container!r}: RSpace does not accept attachments on "
+ f"{'a bench' if target.gid[:2] == 'BE' else 'a sample template'}. "
+ f"Attach to a container, sample, subsample or instrument instead.")
+ result = self._attach(target.gid, file, name)
+ self.invalidate_cache()
+ return result
+
+ def _attach(self, parent_gid: str, file: BinaryIO, name: str) -> dict:
+ """Put ``file`` on the record or field ``parent_gid``, through the Gallery unless
+ ``via_gallery`` is off."""
+ if not self.via_gallery:
+ return self.inv_client.upload_attachment_by_global_id(
+ parent_gid, file, **self._upload_kwargs(name))
+ # No folder id: the Gallery files it into the section its media type belongs to, so
+ # this cannot hit the section-mismatch rule that a chosen folder would.
+ uploaded = self.eln_client.upload_file(file, **self._upload_kwargs(name))
+ # by global id, not through link(): a field global id does not resolve as a path on
+ # its own, because a field is only addressable underneath its record.
+ return self.inv_client.attach_gallery_file_by_global_id(parent_gid, uploaded["globalId"])
+
+ @writes
+ def link(self, path: str, media_file_gid: str) -> dict:
+ """Attach a file already in the Gallery to the record or attachment field at ``path``,
+ without copying the bytes. The attachment reports the Gallery file it points at as
+ ``mediaFileGlobalId``, and removing it leaves the Gallery file alone."""
+ path = self._strip_protocol(path)
+ target = self._resolve(path)
+ if target.kind not in ("record", "shortcut", "field"):
+ raise NotImplementedError(f"{path!r}: a Gallery file links to a record or an attachment field")
+ if target.kind == "field":
+ self._require_empty_field(target, path)
+ result = self.inv_client.attach_gallery_file_by_global_id(target.gid, media_file_gid)
+ self.invalidate_cache()
+ return result
diff --git a/rspace_client/fs/paths.py b/rspace_client/fs/paths.py
new file mode 100644
index 0000000..0abb447
--- /dev/null
+++ b/rspace_client/fs/paths.py
@@ -0,0 +1,142 @@
+"""
+Path grammar helpers shared by the RSpace filesystems.
+
+A path is a slash-delimited sequence of segments. Below a filesystem root each
+segment addresses an RSpace record, and there are three ways of writing one:
+
+ name ``microscope.png`` the record's own name
+ labelled ``microscope.png [GL100]`` name and global ID
+ id ``GL100`` the bare global ID
+
+The *path style* decides which of these a listing produces. A segment carrying a global
+ID is accepted under every style, so a path copied from a labelled or id listing resolves
+anywhere; a bare name only resolves under the ``name`` style, where the lookup exists.
+A segment carrying a global ID resolves in one call and survives a rename. A bare
+name has to be looked up among its parent's children, and where a folder holds two
+records with the same name the listing appends the global ID to the second and
+later ones, so every segment in a listing stays unique and addressable.
+
+Each filesystem adds its own vocabulary on top of this (the Inventory sections, for
+instance); what lives here is only what all of them share.
+"""
+from __future__ import annotations
+
+import re
+from typing import Optional, Tuple
+
+from posixpath import basename
+
+GLOBAL_ID = re.compile(r"^[A-Z]{2}\d+$")
+LABELLED = re.compile(r"^(?P