Skip to content

HASH partition creation reports “already exists”, then immediate open reports “was not found” #98

Description

@hsballoon

Data writing to partitions: 8、60、121、 94.....(as far as we know only 34 is normal)consistently fails.

This is reproducible by calling dldb directly with a single process, single thread, and single session:

from dldb.utils import stable_hash

job_id = "codex-bucket94-probe-1787650993-232"
assert stable_hash(job_id) % 128 == 94

session.add("landing_test", dataframe)

Actual result:

ValueError: Table
'landing_test_type_HASH_column_job_id_partitions_128_partition_94'
already exists

During exception handling, dldb calls open_table(), which then fails with:

ValueError: Table
'landing_test_type_HASH_column_job_id_partitions_128_partition_94'
was not found

Other observations:

  • 483 records have been written successfully to other partitions.
  • Restarting the worker and creating a fresh dldb session does not resolve it.
  • The partition is not visible through table listing/opening.
  • No corresponding partition 94 objects are visible under the S3 prefix.
  • The issue reproduces without WTGatewayClient.

Expected behavior:

dldb should create and open partition 94 successfully, or return a consistent and actionable error explaining why the partition cannot be created.

Environment:

dldb==1.0.0
lancedb==0.34.0
pylance==9.0.0
pyarrow==23.0.1
pandas==2.3.3

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions