Fix logical-table resolution and eliminate redundant catalog scans for cached tables and HASH buckets

[hsballoon]

Problem

LanceSession._open_disk_table() currently resolves a logical table by scanning physical table names with:

if full_table_name.startswith(table_name):

This is unsafe when logical table names share a prefix, for example:

• aaaa
• aaaa_legacy
• aaaa_backup

The selected physical table can depend on the ordering returned by list_tables(). This may cause dldb to infer the wrong partition type and lead to assertion failures or unstable table-opening behavior.

There is also an avoidable performance issue in the HASH table path: opening a partition calls list_tables() before checking whether the physical bucket wrapper is already cached. Repeated queries against the same bucket therefore continue to perform catalog requests even when the table and bucket have already been opened.

These behaviors should be fixed in dldb because dldb owns logical-to-physical table resolution, wrapper lifecycle, catalog access, and cache invalidation. SDK clients should not need to inspect or bypass dldb internals.

Proposal

1. Resolve logical tables exactly

session.filter(logical_table_name, ...) should first perform an exact lookup in the dldb information schema:

record = schema_table.get(table_name)
table = open_table_by_partition_type(
    db_conn,
    schema_table,
    table_name,
    record.partition_type,
    partition,
)

Physical table and bucket discovery should remain inside the corresponding table wrapper. Logical-table resolution must not depend on physical table-name prefix matching.

2. Cache logical-table wrappers in the session

After resolving a logical table, LanceSession should cache and reuse its wrapper for subsequent reads and writes in the same session.

3. Reuse cached HASH bucket wrappers without catalog scans

When a requested HASH bucket is already present in the wrapper’s physical-bucket cache, dldb should return it before calling list_tables().

A catalog scan should only be necessary when the requested bucket has not yet been opened or its cached entry has been invalidated.

4. Invalidate caches on catalog mutations

Creating, dropping, replacing, or recreating a logical or physical table should automatically invalidate the affected logical-table and physical-bucket cache entries.

Callers should not need to manually clear internal dldb caches after operations performed through dldb.

Expected behavior

• Logical tables are always resolved using exact information-schema records.
• Overlapping logical table-name prefixes cannot affect resolution.
• Repeated operations reuse the same logical-table wrapper.
• Repeated access to an already opened HASH bucket does not call list_tables().
• Catalog mutations cannot leave stale wrappers or bucket objects in use.
• Existing snapshot and checkout_latest behavior remains unchanged.

Please add regression tests covering:

• Logical tables with overlapping prefixes.
• Overlapping names using different partition types.
• Repeated access to the same logical table and HASH bucket.
• Cache invalidation after create, drop, and replace operations.

[Fantasy97]

Implemented on feat/add-dldb-package.

Logical-table resolution now uses an exact information_schema lookup. _open_disk_table() prefix matching on physical names is gone. session.filter("aaaa") can no longer open aaaa_legacy. Missing schema records fail immediately (no reload, no physical-name fallback). Physical table names are not a session API.

Catalog scans: HASH/VALUE open_table() returns cached buckets without list_tables(). HASH filter with explicit partitions skips list_partitions() when every requested bucket is already cached. Unmaterialized HASH buckets are still treated as empty. list_partitions() still lists the catalog when it needs to discover buckets.

Invalidation stays same-session: full drop pops the wrapper, drop-partition pops that bucket, create_table replaces the wrapper. No cross-process invalidation.

Tests in tests/dldb/test_logical_table_resolution.py: overlapping prefixes (including different partition types), repeated HASH bucket access with no catalog scan, drop/recreate, and rejecting a physical table name.

[hsballoon]

Thanks for the fix. I verified the current feat/add-dldb-package branch at 9064f02.

The fix works for our HASH-partitioned landing and serving tables: logical-table resolution is exact, and repeated access to an already opened HASH bucket no longer calls list_tables().

However, the fix currently breaks unpartitioned tables, which we also use in some scenarios.

An unpartitioned table works in the session that creates it, but a new session cannot reopen it:

import tempfile

import dldb
import pandas as pd
import pyarrow as pa

path = tempfile.mkdtemp()
schema = pa.schema([pa.field("id", pa.int64())])

session = dldb.connect(path)
session.create_table("simple", schema)
session.add("simple", pd.DataFrame({"id": [1]}))

other = dldb.connect(path)
other.filter("simple", query="id IS NOT NULL")

The last call fails with:

AssertionError: Partition column must be specified

The old dldb-v1.0.0 implementation successfully returns the row in the same scenario.

The new exact-resolution path appears to select a partitioned table wrapper before checking that record.partition_column is empty. As a result, the current fix works for HASH-partitioned tables but does not correctly reopen unpartitioned tables.

The branch also removes dldb.table.open_table, which our checkpoint code previously used to open an unpartitioned table. We can update the SDK to stop using that method and rely on the new exact-resolution path, but the replacement path must first support unpartitioned tables correctly. Fixing only the SDK call would not solve other normal dldb operations such as session.filter() on other unpartitioned tables.