Skip to content

chore/tooling modernization - #69

Open
nebay-abraha wants to merge 21 commits into
mainfrom
chore/tooling-modernization
Open

nebay-abraha wants to merge 21 commits into
mainfrom
chore/tooling-modernization

Conversation

@nebay-abraha

@nebay-abraha nebay-abraha commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Description

This PR moves the project metadata to PEP 621 and Poetry 2, drops Python 3.9, adds ruff and mypy with a CI lint job, and adds a release workflow that publishes to PyPI when a v* tag is pushed. Running ruff and mypy over the code turned up six bugs, and each fix has its own commit.

Please review commit by commit. The style: commit 8c991e4 is unedited ruff output, so you can skim or skip it.

Breaking changes

Change Who is affected
Python 3.9 is no longer supported. requires-python is now >=3.10. Users still on 3.9
Sample.wherep() returns a list, as documented, instead of a set. Code that calls .add() on the result, uses set operators on it, or compares it with == {...}
GridContainerPost sends cType: "GRID" instead of "LIST". Anyone who relied on grid containers being created as list containers. This is unlikely to be intentional.

CHANGELOG.md lists all three under "Potentially breaking".

Changes

Packaging (0358362, 45e1535)

  • Project metadata moves from [tool.poetry] to the standard PEP 621 [project] table. Dev dependencies move to [dependency-groups], and the build backend is poetry-core 2.x.
  • CI uses Poetry 2.4.1, because Poetry 1.8 cannot read the [project] table.
  • poetry.lock is regenerated. main added fsspec and universal-pathlib to pyproject.toml without updating the lock file, so poetry check --lock failed after the rebase. The new lock adds fsspec, pathlib-abc and universal-pathlib and changes no existing pins.

Linting, formatting and typing (05c9696, 8c991e4, bc82e3a)

  • ruff replaces black for linting and formatting. mypy, pre-commit and pytest-cov are new dev dependencies, and all four tools are configured in pyproject.toml. ruff and mypy skip notebook_sync.
  • 8c991e4 is the output of ruff check --fix followed by ruff format, with no manual edits.
  • bc82e3a fixes by hand what ruff could not fix automatically, and adds the annotations mypy needs to pass. These include PEP 604 unions, X | None for parameters that default to None, dict[str, Any] on self.data, and a ClientBase._get_api_url() stub. It also changes three behaviours:
    • Container.capacity() raises NotImplementedError instead of returning None.
    • The EXACT grid move zips items and locations with strict=True, so a length mismatch raises an error instead of silently dropping the extra items.
    • In examples/tree_upload.py, a bare exit did nothing. It is now sys.exit(1).

Bug fixes

  • dbd387f removes shared mutable default arguments. Several inv and eln methods had defaults such as Pagination(), tags=[], extra_fields=[], TopLevelTargetLocation(), item_ids=[] and typesToInclude=[]. Python creates a default once and every call that omits the argument shares it, so a SearchFilter merged into the default Pagination leaked into later calls. These parameters now default to None, and each call builds a fresh value.
  • f863a30 makes Sample.wherep() return a list, matching its annotation and docstring.
  • 598073d makes GridContainerPost send cType: "GRID".
  • 6cfdc59 adds the missing self to ListContainer.is_list(). Without it, calling the method on an instance raised TypeError.
  • f581c5b fixes the URL validator. Its error message was missing the f prefix and printed the literal {...} placeholders. It also used a bare except, which caught everything including KeyboardInterrupt. It now catches the ValueError that urlparse raises and chains it.
  • 8323263 restores the server's error detail when an ELN export job fails. The + self._get_formated_error_message(...) line had no surrounding parentheses, so Python treated it as a separate statement and threw the result away. The error read only "Export job failed: ". ad8ce4f adds a test for this.

CI and release (83036f8, 6326a72)

  • A new lint job runs ruff check, ruff format --check and mypy rspace_client. The unit test job now waits for it.
  • The unit test matrix covers Python 3.10 to 3.13, up from 3.9 to 3.12. The integration job runs on 3.13.
  • The unit tests report coverage and fail if it drops below 55%.
  • The new release.yml workflow runs when a v* tag is pushed. It runs the unit tests, checks that the tag matches the version in pyproject.toml, builds the package and publishes it to PyPI. Publishing uses Trusted Publishing, which authenticates with OIDC, so no PyPI API token is stored anywhere.
  • release.sh now takes only the new version number. It bumps the version with poetry version, formats with ruff, then commits and tags. CI does the publishing.
  • Dependabot opens one grouped pull request a week for pip updates and one for GitHub Actions updates.

Tests (1213f8e, 32a8f3c, ad8ce4f)

  • Newer RSpace servers treat the DataCite password as write-only and return null for it. The DataCite settings test no longer asserts that the server echoes the password back.
  • The four DataCite tests used to overwrite the server's settings with dummy credentials and then fail to restore them. On a server with DataCite enabled, the restore passed password=None and raised ValueError. On a server with saved but disabled credentials, it left the dummy values behind. The tests now skip when DataCite is enabled or when real credentials are saved, and they always disable DataCite when they finish. CI and dev servers with no DataCite configuration still run them.
  • ad8ce4f adds a test for FAILED export jobs and moves the conda environment to Python 3.12.

Docs (3588123)

  • Updates the Unreleased section of CHANGELOG.md, DEVELOPING.md and the comments in pyproject.toml.

How to test

poetry install
poetry run ruff check rspace_client examples
poetry run ruff format --check rspace_client examples
poetry run mypy rspace_client
poetry run pytest -m "not integration" --cov=rspace_client 

The integration tests still need a live RSpace server. Run them with poetry run pytest -m integration and the usual environment variables.

🤖 Generated with Claude Code

nebay-abraha and others added 15 commits October 31, 2026 13:13
Move project metadata and dependencies from [tool.poetry] to the standard
[project] table and dev dependencies to [dependency-groups], with
poetry-core 2.x as the build backend. Python 3.9 is dropped.

CI moves to Poetry 2.4.1 (1.8 cannot read the [project] table) and the
unit test matrix becomes 3.10-3.13; the integration job runs on 3.13.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace black with ruff (lint + format) and add mypy, pre-commit and
pytest-cov as dev dependencies, with their configuration in
pyproject.toml. notebook_sync is excluded from ruff and mypy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Several methods used mutable or stateful defaults (Pagination(),
tags=[], extra_fields=[], TopLevelTargetLocation(), item_ids=[],
typesToInclude=[]). Those objects are created once and shared by every
call that omits the argument, so e.g. a SearchFilter merged into the
default Pagination leaked into later calls. Default to None and build a
fresh value per call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wherep() is annotated and documented as returning a list of breadcrumb
strings but returned the de-duplicating set directly.

BREAKING CHANGE: callers relying on set behaviour (.add(), set
operators, == {...}) must be updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GridContainerPost set cType to "LIST", so grid containers were posted
with the list container type.

BREAKING CHANGE: grid containers are now sent with cType "GRID".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
is_list() was declared without self, so calling it on an instance
raised TypeError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The TypeError message lacked its f prefix, so it printed the literal
"{type(item)} {item!r}" placeholders, and the bare except swallowed
any exception (including KeyboardInterrupt). Catch the ValueError that
urlparse raises and chain it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The "+ self._get_formated_error_message(...)" continuation sat on its own
line without parentheses, so it was evaluated as a separate, discarded
expression and msg was only "Export job failed: ".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Purely mechanical output of `ruff check --fix` followed by `ruff format`
over rspace_client, examples and .github/scripts. No manual edits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Hand fixes for what `ruff check --fix` could not auto-fix, plus the type
annotations and small refactors needed for `mypy rspace_client` to pass:
PEP 604 unions, `X | None` for None-defaulted parameters, dict[str, Any]
annotations on self.data, and a ClientBase._get_api_url() stub.

Small behaviour changes worth noting:
- Container.capacity() raises NotImplementedError instead of returning
  None.
- The EXACT grid move zips items and locations with strict=True.
- examples/tree_upload.py: a bare `exit` (a no-op) is now sys.exit(1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New lint job runs ruff check, ruff format --check and mypy; unit tests
now depend on it and report coverage, failing below 55%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ease.sh

Pushing a v* tag runs release.yml: unit tests, a tag/pyproject version
check, poetry build, then publish via PyPI Trusted Publishing (OIDC, no
API token). release.sh now takes only the new version, bumps it with
`poetry version`, formats with ruff instead of black, commits and tags,
and leaves publishing to CI. Dependabot groups weekly pip and GitHub
Actions updates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Newer RSpace servers treat the DataCite password as write-only and
return null for it from the settings update endpoint, so
test_update_datacite_settings_enabled failed with
`None != 'test_password'`. Drop that one assertion; the other fields
are still checked, and the test now passes against both older servers
that echo the password and newer ones that don't.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main added fsspec (runtime) and universal-pathlib (dev) to pyproject.toml
without updating the lock file, so `poetry check --lock` failed after the
rebase. Regenerated with `poetry lock`, which only adds fsspec,
pathlib-abc and universal-pathlib; no existing pins changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ings

The four DataCite tests write dummy credentials and then tried to restore
the original settings from get_datacite_settings(). The API no longer
returns the stored password, so on a server with DataCite enabled the
restore passed password=None and raised ValueError, and on one with
saved-but-disabled credentials it left the dummies in their place.

Skip these tests when DataCite is enabled, or when a username other than
the tests' own dummy ones is saved, and always clean up by disabling
DataCite. CI and dev servers with no DataCite configuration still run
them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The FAILED export-job path lacks regression coverage, and the documented conda setup still creates an unsupported Python 3.7 environment instead of Python 3.10+.

2 open issues
What changed in this PR

Modernizes packaging and development tooling for the RSpace ELN and Inventory client, alongside Python syntax cleanup and targeted behavior fixes.

Changes:

  • Migrates package metadata to the standard [project] format and raises the minimum Python version to 3.10.
  • Adds Ruff, mypy, pre-commit, coverage checks, automated PyPI publishing, and Dependabot.
  • Updates client code, tests, and examples, including safer defaults, clearer errors, and DataCite test safeguards.
File Description
rspace_client/​validators.py Corrects URI errors and simplifies membership checks.
rspace_client/​utils.py Normalizes import spacing.
rspace_client/​tests/​validator_test.py Cleans up imports and headers.
rspace_client/​tests/​template_builder_test.py Formats tests and removes an unused assignment.
rspace_client/​tests/​temperature_test.py Cleans up imports and headers.
rspace_client/​tests/​sample_post_test.py Removes the redundant encoding header.
rspace_client/​tests/​quantity_unit_test.py Normalizes imports and headers.
rspace_client/​tests/​mock_server_case.py Modernizes shared-server type annotations.
rspace_client/​tests/​mock_rspace/​server.py Modernizes typing, formatting, and exception chaining.
rspace_client/​tests/​invapi_test.py Adds DataCite safeguards and cleans up tests.
rspace_client/​tests/​inv_lom_test.py Normalizes imports and headers.
rspace_client/​tests/​inv_attachment_fs_test.py Standardizes imports, quoting, and formatting.
rspace_client/​tests/​id_test.py Cleans up imports and headers.
rspace_client/​tests/​http_layer_test.py Modernizes formatting and context managers.
rspace_client/​tests/​grid_placement_test.py Cleans up imports and headers.
rspace_client/​tests/​grid_container_test.py Normalizes imports and file opening.
rspace_client/​tests/​galaxy_plugin_contract_test.py Formats existing compatibility tests.
rspace_client/​tests/​fs_workspace_test.py Formats workspace tests and context managers.
rspace_client/​tests/​fs_review_test.py Formats filesystem regression tests.
rspace_client/​tests/​fs_name_paths_test.py Formats path-resolution tests and fixtures.
rspace_client/​tests/​fs_inventory_tree_test.py Formats inventory filesystem tests.
rspace_client/​tests/​fs_integration_test.py Formats live filesystem integration tests.
rspace_client/​tests/​field_content_test.py Cleans up imports and an expression.
rspace_client/​tests/​exceptions_test.py Formats error-response fixtures and assertions.
rspace_client/​tests/​elnapi_test.py Normalizes imports and test formatting.
rspace_client/​tests/​client_methods_test.py Formats request assertions and context managers.
rspace_client/​tests/​base_test.py Normalizes imports and dotenv initialization formatting.
rspace_client/​inv/​template_builder.py Modernizes typing and clarifies URI errors.
rspace_client/​inv/​sample_builder2.py Normalizes imports and headers.
rspace_client/​inv/​quantity_unit.py Corrects the unit-label return annotation.
rspace_client/​inv/​attachment_fs.py Modernizes legacy filesystem typing and syntax.
rspace_client/​fs/​tree.py Modernizes tree-helper typing and formatting.
rspace_client/​fs/​rspace.py Modernizes unified filesystem typing and formatting.
rspace_client/​fs/​paths.py Modernizes path-helper annotations and formatting.
rspace_client/​fs/​inventory.py Modernizes inventory filesystem typing and formatting.
rspace_client/​fs/​__init__.py Normalizes filesystem imports.
rspace_client/​fs_utils.py Normalizes quoting and docstring indentation.
rspace_client/​exceptions.py Modernizes optional type annotations.
rspace_client/​eln/​fs.py Modernizes legacy gallery syntax and typing.
rspace_client/​eln/​filetree_importer.py Updates annotations and simplifies expressions.
rspace_client/​eln/​field_content.py Simplifies filtering and normalizes imports.
rspace_client/​eln/​eln.py Replaces shared defaults and repairs export failure details.
rspace_client/​eln/​dcs.py Cleans up header spacing.
rspace_client/​eln/​advanced_query_builder.py Reorders imports.
rspace_client/​client_base.py Improves typing, exception chaining, and subclass contract.
rspace_client/​__init__.py Normalizes package imports and docstring formatting.
release.sh Updates versioning and tag-based release preparation.
README.md Documents the Python 3.10 minimum.
pyproject.toml Migrates metadata and configures development tools.
examples/​tree_upload.py Modernizes syntax and fixes error-path termination.
examples/​status.py Simplifies optional status-field handling.
examples/​share_documents.py Clarifies a variable and cleans up syntax.
examples/​search_documents_by_form.py Normalizes imports and removes obsolete compatibility code.
examples/​paging_through_users.py Modernizes logging and narrows exception handling.
examples/​paging_through_results.py Removes obsolete print compatibility code.
examples/​inventory_link_field.py Cleans up headers and spacing.
examples/​import_word_file.py Normalizes imports and removes obsolete compatibility code.
examples/​import_inventory_csv.py Simplifies CSV string construction.
examples/​import_directory.py Separates and normalizes imports.
examples/​get_activity.py Modernizes imports and output strings.
examples/​freezer.py Cleans up imports, unused names, and formatting.
examples/​export_records.py Modernizes output formatting.
examples/​download_attachments.py Normalizes imports and removes obsolete compatibility code.
examples/​create_sample.py Normalizes headers and formatting.
examples/​create_instrument.py Normalizes imports and headers.
examples/​create_form.py Removes obsolete print compatibility code.
examples/​create_folder_and_notebook.py Modernizes output and removes compatibility code.
examples/​create_document.py Cleans up imports and compatibility code.
DEVELOPING.md Documents updated setup, checks, and publishing.
CHANGELOG.md Records tooling changes and compatibility notes.
.pre-commit-config.yaml Adds Ruff and mypy hooks.
.github/​workflows/​release.yml Adds tested builds and PyPI Trusted Publishing.
.github/​workflows/​codeql-and-tests.yml Adds linting, coverage checks, and updated Python versions.
.github/​scripts/​warmup_sysadmin.py Adds spacing before an existing comment.
.github/​dependabot.yml Schedules grouped dependency and Actions updates.

Review effort: Balanced

Comment thread DEVELOPING.md Resolved
Comment thread rspace_client/eln/eln.py Resolved
nebay-abraha and others added 2 commits October 31, 2026 14:27
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nebay-abraha nebay-abraha changed the title Chore/tooling modernization chore/tooling modernization Oct 31, 2026
Poetry 1.x generated the "Programming Language :: Python" classifiers
from the python constraint. Poetry 2 doesn't generate them when the
metadata is in the [project] table, so the 2.8.0 wheel would have had
none. That would also break the README's PyPI "Python versions" badge.
List 3.10 to 3.13 by hand to match the CI matrix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Regression tests are needed to verify filter isolation across filtered-then-unfiltered calls to both list_samples() and stream_samples(), preventing the shared-default bug from recurring unnoticed.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Add regression tests for filter isolation between calls

rspace_client/​inv/​inv.py:1425

The existing pagination tests do not check that filters are isolated between calls, so they would miss a return of the shared-default bug. Add an offline regression test that calls list_samples(sample_filter=SearchFilter(owned_by="alice")) and then list_samples() without pagination, asserting that the second request has neither ownedBy nor deletedItems. Cover the same filtered-then-unfiltered sequence for stream_samples() with default pagination.

2 resolved since last review

Review effort: Balanced

@nebay-abraha
nebay-abraha marked this pull request as ready for review November 2, 2026 06:06

@tilorspace tilorspace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the tooling modernisation. Locally on 3173991, all 349 unit tests pass, ruff check, ruff format --check and mypy rspace_client are clean, and coverage is 73.9%. I compared the ASTs before and after the style: commit 8c991e4 and found no behaviour changes, only import order, f-strings and unions. The FAILED-export fix matches the server: rspace-web's JobsApiHandlerImpl puts an ApiError with message and errors in result for FAILED jobs.

Inline comments below, roughly most important first. One more that sits outside the diff:

rspace_client/inv/inv.py:150 (AutoFit), missing f-prefix. The second half of the out-of-range message is a plain string:

+ " must fit in dimensions ({total_columns}, {total_rows}"

so ByRow(5, 1, 3, 3, ss) prints the literal {total_columns}, {total_rows}. This is the same bug fixed in validators.py in f581c5b. The selected ruff rules don't catch it (RUF027 would, once it leaves preview).

🤖 Generated with Claude Code

Comment thread rspace_client/eln/eln.py Resolved
Comment thread rspace_client/inv/inv.py Resolved
Comment thread rspace_client/tests/invapi_test.py Resolved
Comment thread release.sh Resolved
Comment thread rspace_client/inv/inv.py Resolved
Comment thread rspace_client/inv/inv.py Resolved
Comment thread .github/dependabot.yml Resolved
Comment thread .github/workflows/release.yml Resolved
Comment thread .github/workflows/codeql-and-tests.yml Resolved
- eln: abandoned or malformed export jobs raise ApiError and are logged
- inv: list/stream calls no longer write the filter into the caller's Pagination
- inv: Workbench.capacity() returns sys.maxsize; fix grid error message
- inv: extra_fields=None sends [] like the default
- tests: DataCite tests reuse their own leftover settings
- release: release.sh runs only on an up-to-date main; one Unreleased heading
- ci: release.yml checks the tag is on main and runs ruff/mypy; coverage 70%
- pre-commit: hooks use the Poetry-pinned ruff/mypy; Dependabot updates them separately

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nebay-abraha

Copy link
Copy Markdown
Contributor Author

Addressed @tilorspace's review in d51a87f: all inline comments plus the AutoFit f-prefix. Replies are on each thread.

Also from a follow-up review: export polling no longer crashes on errors: null or a missing percentComplete, and DEVELOPING.md describes the new release flow.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Providers with test usernames can bypass the guard even when enabled or targeting another server, allowing tests to overwrite credentials and leave them disabled without restoring their write-only passwords.

1 open issue

Review effort: Balanced

Comment thread rspace_client/tests/invapi_test.py Resolved
The DataCite integration tests overwrite the server's settings, and the
API never returns the stored password, so real settings cannot be
restored. The guard trusted the username alone, so a real account named
test_user, or one on a real server, would be reused and overwritten.

Reuse saved settings only when the username, server URL and repository
prefix all match what the tests write, or when the settings are blank
(disabled, no username or prefix, password ). Anything else, including
any stored secret, skips the tests. A leftover from an interrupted run
still matches, so recovery keeps working.

The logic moves to datacite_settings_conflict(), with offline tests for
blank, leftover, test username on a real server, other username, stored
secret without a username, and saved credentials while disabled.
test_get_datacite_settings now also asserts the password is None or ,
the response shape the guard relies on.

Verified against a local server: the five DataCite integration tests
pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants