Skip to content

Fix get_files(directory) on S3 and local storage drivers - #222

Open
tmgbedu wants to merge 1 commit into
mainfrom
fix/get-files-directory-218
Open

Fix get_files(directory) on S3 and local storage drivers#222
tmgbedu wants to merge 1 commit into
mainfrom
fix/get-files-directory-218

Conversation

@tmgbedu

@tmgbedu tmgbedu commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Fixes #218

Problem

get_files(directory) was broken on both drivers, in two different ways, and the drivers disagreed about what the method means — making it unusable in disk-agnostic code:

  1. S3 — the nested-file filter tested "/" not in key against the whole key instead of the part below the prefix. Any key matched by a non-empty Prefix necessarily contains prefix + '/', so every match was discarded and root-level keys leaked through instead. get_files("backups") always returned [].
  2. Localos.listdir yields bare names, but contents were fetched with self.get(f), which resolves against the disk root rather than root/<directory>. The lookup missed, get() swallowed the FileNotFoundError, and callers silently received File objects with None content.
  3. FakeFakeDriver extends LocalDriver, so it inherited the same broken listing (its own assert_count(n, directory) relied on it).

Fix

One converged, documented contract across all three drivers: non-recursive listing of the files directly under the given directory, bare filenames, no directory entries, [] for empty or nonexistent directories.

  • S3: filter with a slash-terminated prefix (Prefix="backups/") and test only the key's relative part. This also stops get_files("backups") from matching a backups-old/ sibling and skips directory-marker keys.
  • Local: read contents from join(directory, f); return [] for a nonexistent directory (matching S3's behavior for an unknown prefix).
  • Fake: inherits the fixed LocalDriver.get_files; dedicated tests lock the semantics in.

Testing

  • Mocked S3 unit tests for the prefix/relative-key logic (root listing, trailing slash, partial-name prefix, directory markers, nested dirs).
  • Live integration tests against a real S3 API (MinIO in Docker) — bucket with keys at root, one and two levels deep; asserts every listing shape. Auto-skipped when no MinIO endpoint is reachable, endpoint/credentials overridable via MINIO_ENDPOINT / MINIO_ACCESS_KEY / MINIO_SECRET_KEY. All 8 pass against the live server.
  • Local/Fake tests now assert file content equals the bytes written, not merely non-None, plus trailing-slash, empty-dir, nonexistent-dir and nested-dir cases.
  • boto3 added to the dev dependency group to support the S3 tests.

Full suite: uv run pytest --ignore=tests/masoniteorm/postgres --cov2213 passed, 7 skipped, coverage 84.50% (threshold met).

🤖 Generated with Claude Code

https://claude.ai/code/session_01KVFQTBaxKHnCZfjmtx2HYg

Fixes #218. get_files(directory) was broken differently on each driver,
and the two disagreed about what the method means:

- S3: the nested-file filter tested "/" against the whole key instead of
  the part below the prefix, so any non-empty directory returned [] while
  root-level keys leaked through. Filter now uses a slash-terminated
  Prefix and tests only the key's relative part, which also stops
  'backups' from matching 'backups-old/'.
- Local: os.listdir yields bare names but contents were fetched via
  get(name), which resolves against the disk root — a miss that get()
  swallows, returning File objects with None content. Contents are now
  read from directory/name, and a nonexistent directory returns [].
- Fake: inherits the fixed LocalDriver.get_files; tests lock the
  semantics in.

All three drivers now share one contract: non-recursive listing of the
files directly under the given directory, bare filenames, no directory
entries, [] for empty or nonexistent directories.

Adds mocked S3 unit tests plus live integration tests against a real S3
API (MinIO, skipped when unreachable), and boto3 as a dev dependency to
support them.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KVFQTBaxKHnCZfjmtx2HYg
@codecov

codecov Bot commented Sep 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

get_files(directory) is broken on both the S3 and local drivers

1 participant