[TZF-260079] Add admin-only LanceDB explorer by DhwaniKhandelwal-tech · Pull Request #82 · recursivezero/tz-fabric

DhwaniKhandelwal-tech · 2026-07-29T16:51:55Z

Summary

Closes #79

Adds a secure, read-only LanceDB Explorer that allows administrators to inspect LanceDB tables, schemas, metadata, paginated rows and vector details from a private application route.

Screenshots

Administrator access gate

LanceDB table overview

Row and vector details

Backend changes

Added protected administrator endpoints:
- GET /api/v1/admin/lancedb/tables
- GET /api/v1/admin/lancedb/{table_name}
- GET /api/v1/admin/lancedb/{table_name}/rows
- GET /api/v1/admin/lancedb/{table_name}/rows/{row_id}
Added administrator-secret validation through X-Internal-Secret
Added table-name and query-parameter validation
Added server-side pagination
Added exact tag filtering
Added scalar-column sorting
Added lazy full-vector retrieval
Added safe metadata and non-finite value serialisation
Kept all explorer operations read-only

Frontend changes

Added private route: /admin/lancedb
Added administrator access gate
Added table selector and refresh controls
Added schema, metadata, row-count and vector-dimension panels
Added paginated row grid
Added tag filtering and sorting
Added row-detail and full-vector viewer
Added copy actions for row JSON, image URI, hash and vector
Added loading, empty, denied-access and backend-error states
Kept the route out of public navigation

Security

Administrator endpoints require X-Internal-Secret
The entered secret is retained only in page memory
Production requires an explicit INTERNAL_API_KEY
No create, update or delete operations are exposed

Validation

Backend

Black: passed
Flake8: passed
Mypy: passed
Ruff: passed
Unit and integration tests: 25 passed

Frontend

Biome: passed
ESLint: passed
Vitest: 9 test files passed
Vitest: 39 tests passed
Production build: passed

Merge target

This PR is raised against develop.

…/tz-fabric into feature/TZF-260079

DhwaniKhandelwal-tech · 2026-07-29T17:00:32Z

Screenshots added:

recursivezero · 2026-07-30T15:11:57Z

I was able to see the DB explorer with the password and when logged in and there was error on the page ; no table display and when click on try again; i got exited and then not able to logged back with same password
getting 403 error for API http://localhost:5173/api/v1/admin/lancedb/tables

check that what and how do we handle if table not exist

recursivezero · 2026-07-30T15:17:13Z

why do we have duplicate APIs documentation on localhost:8002

and what is the admin/login API signature?

recursivezero · 2026-07-30T15:25:07Z

Also I see there are changes in requirements.txt file means some packages getting added , please add using poetry as we are using poetry not requirements.txt

recursivezero · 2026-08-01T17:19:00Z

do not call lance db directly when user logged in instead give a place to add path of lancedb and then it will read and scan

recursivezero · 2026-08-04T16:07:18Z

recursivezero · 2026-08-04T16:10:05Z


+const localSource = {
+  storage: "local" as const,
+  location: "C:\\data\\fabric.lancedb",


why do we fix this path?

…ation

DhwaniKhandelwal-tech · 2026-08-21T07:06:34Z

LanceDB Admin Explorer — Setup & Usage

This PR adds the LanceDB Admin Explorer for inspecting LanceDB databases from:

Local LanceDB storage
Amazon S3
Cloudflare R2

The explorer is designed to be usable by any developer/team member after configuring the required backend environment variables. No machine-specific paths or credentials are hardcoded.

Important: AWS/R2 credentials must only be configured on the backend. They should never be entered in the admin UI, frontend environment, or committed to Git.

1. Backend Environment Setup

From the repository:

cd backend

For local development, make sure the backend environment file exists.

For example:

cp .env.sample .env.development

Configure the following values as required.

Common configuration

ENVIRONMENT="development"
API_PREFIX="/api/v1"
PORT=8002
Used to authenticate the LanceDB Admin Explorer.
Choose a secure value for the environment.
INTERNAL_API_KEY="<your-admin-secret>"
ALLOW_LOCAL_ORIGINS="true"

The value configured in:

INTERNAL_API_KEY

is the secret that must be entered on the LanceDB Explorer login page.

It is separate from AWS/R2 credentials.

2. Local LanceDB Configuration

Local mode does not require AWS/R2 credentials.

The backend uses the project's configured LanceDB database directory:

backend/database

The expected structure is similar to:

backend/

└── database/

├── table-a.lance/

├── table-b.lance/

└── ...

To check which local LanceDB tables are currently available:

find backend/database -type d -name "*.lance"

Example:

backend/database/tz-fabric-table.lance

Important

Do not provide the complete backend/ directory to the explorer.

The explorer connects to the LanceDB database directory and then discovers the tables contained inside it.

Therefore:

backend/                      ❌

backend/database/             ✅ database location

backend/database/table.lance  ✅ physical table

The user does not need to manually enter the local path in the admin UI.

Select:

Storage → Local database

and click:

Scan tables

The backend will use the configured local database location and return the available tables.

3. Amazon S3 Setup

To enable Amazon S3 support, configure these values in the backend environment:

AWS_ACCESS_KEY_ID="<aws-access-key>"

AWS_SECRET_ACCESS_KEY="<aws-secret-key>"

AWS_REGION="<aws-region>"

AWS_BUCKET_NAME="<bucket-name>"

Example region:

AWS_REGION="ap-south-1"

Credentials stay entirely on the backend.

S3 — Database at bucket root

If the LanceDB database is stored directly at the configured bucket root:

bucket/

├── table-a.lance/

└── table-b.lance/

then in the Admin Explorer:

Storage → Amazon S3

Leave the database-location field empty and click:

Scan tables

The backend will use:

AWS_BUCKET_NAME

automatically.

S3 — Database inside a prefix/folder

If the layout is:

bucket/

└── lancedb/

├── table-a.lance/

└── table-b.lance/

enter:

s3://<bucket-name>/lancedb

Example:

s3://my-bucket/lancedb

Do not enter AWS credentials in this field.

4. Cloudflare R2 Setup

Cloudflare R2 uses an S3-compatible API, which is why LanceDB locations still use the s3:// URI format.

Configure R2 on the backend:

R2_ACCESS_KEY_ID="<r2-access-key>"

R2_SECRET_ACCESS_KEY="<r2-secret-key>"

R2_ACCOUNT_ID="<cloudflare-account-id>"

R2_BUCKET_NAME="<r2-bucket-name>"

R2_ENDPOINT="https://<account-id>.r2.cloudflarestorage.com"

R2_REGION="auto"

If used elsewhere by the application, the CDN URL can also be configured:

R2_PUBLIC_CDN="https://<your-cdn-domain>"

R2_PUBLIC_CDN is not required for connecting the LanceDB Explorer to R2.

Important distinction

This:

https://<account-id>.r2.cloudflarestorage.com

is the R2 API endpoint.

It belongs in:

R2_ENDPOINT

It should not be entered into the Admin Explorer's database-location field.

5. R2 — Database at bucket root

If R2 contains:

R2 bucket/

├── table-a.lance/

├── table-b.lance/

└── ...

select:

Storage → Cloudflare R2

Leave the database-location field empty and click:

Scan tables

The backend automatically uses:

R2_BUCKET_NAME

R2_ENDPOINT

R2_ACCESS_KEY_ID

R2_SECRET_ACCESS_KEY

R2_REGION

No Cloudflare credentials need to be provided through the browser.

6. R2 — Database inside a prefix/folder

For an R2 structure such as:

R2 bucket/

└── databases/

└── fabric/

├── table-a.lance/

└── table-b.lance/

enter the LanceDB database location as:

s3://<r2-bucket-name>/databases/fabric

Then click:

Scan tables

Do not enter:

https://<account-id>.r2.cloudflarestorage.com/...

The HTTPS endpoint is backend configuration, not a database location.

7. Why R2 Uses an `s3://` Location

R2 provides an S3-compatible object-storage API.

Therefore the database location is represented as:

s3://bucket/database-prefix

while the backend separately supplies the R2-specific endpoint and credentials.

Conceptually:

Admin Explorer

|

| s3://bucket/database-prefix

v

Backend

|

+-- R2_ENDPOINT

+-- R2_ACCESS_KEY_ID

+-- R2_SECRET_ACCESS_KEY

+-- R2_REGION=auto

|

v

Cloudflare R2

This allows R2 credentials to remain server-side.

8. Install and Run the Backend

From:

cd backend

install dependencies:

poetry install --with dev

Start the backend using the project's development command:

poetry run fabric dev

The development API should normally be available at:

http://localhost:8002

Verify the API:

curl http://localhost:8002/api/v1/health

The LanceDB admin endpoint can also be checked with:

curl -i http://localhost:8002/api/v1/admin/lancedb/access

A 403 without an admin secret is expected and confirms that the endpoint exists.

9. Install and Run the Frontend

From another terminal:

cd frontend

npm ci

npm run dev

The development frontend will normally run at:

http://localhost:5173

Open:

http://localhost:5173/admin/lancedb

The frontend development server proxies /api/v1 requests to the backend development server.

10. Login to LanceDB Explorer

On the first page, enter the value configured as:

INTERNAL_API_KEY

Then click:

Open explorer

This only verifies administrator access.

It does not automatically scan any database.

AWS and R2 credentials must never be entered here.

11. Using the Explorer

After authentication:

Local

Storage → Local database

Scan tables

The server scans the configured local LanceDB database directory.

Amazon S3

For configured bucket root:

Storage → Amazon S3

Location → leave empty

Scan tables

Or specify a database prefix:

s3://bucket/database-prefix

Cloudflare R2

For configured bucket root:

Storage → Cloudflare R2

Location → leave empty

Scan tables

Or specify a database prefix:

s3://bucket/database-prefix

12. Table Selection

The supplied location represents a LanceDB database, not an individual backend source file.

After scanning, the API returns the tables found at that location.

For example:

database/

├── users.lance/

├── documents.lance/

└── embeddings.lance/

The explorer then provides the available tables through the Table selector.

Therefore users should provide the database location and select the required table from the UI instead of browsing arbitrary backend files.

13. Available Explorer Functionality

After a table is selected, the admin explorer can be used to inspect the table and its data, including available information such as:

Table list
Table/schema information
Row count
Schema metadata
Embedding/vector information
Vector dimensions
Table rows
Pagination
Page size
Tag filtering
Scalar-field sorting
Row details
Vector inspection
Rescan
Change storage source
Lock explorer

The explorer is intended as a read-only administrative inspection tool.

It does not expose destructive database operations such as deleting tables/rows, vacuuming, or modifying production records.

14. Credentials & Security

Credentials must be supplied using backend environment configuration only.

Do not commit:

AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY R2_ACCESS_KEY_ID R2_SECRET_ACCESS_KEY

INTERNAL_API_KEY

Do not expose them through:

frontend/.env

browser/local storage

Admin Explorer inputs

GitHub commits

PR descriptions/comments

screenshots

Each developer/environment should configure its own backend .env.development or deployment-secret configuration.

Quick Reference

Storage	Admin-page location	Backend configuration
Local	No path required	Uses configured local DATABASE_PATH
S3 bucket root	Leave empty	AWS_BUCKET_NAME + AWS credentials
S3 prefix	s3://bucket/path	AWS credentials
R2 bucket root	Leave empty	R2_BUCKET_NAME + R2_ENDPOINT + R2 credentials
R2 prefix	s3://bucket/path	R2 endpoint + R2 credentials

Example

Local:
Storage = Local database
Location = N/A
S3:

Storage = Amazon S3

Location = s3://my-bucket/lancedb
R2:

Storage = Cloudflare R2

Location = s3://my-r2-bucket/lancedb

For both S3 and R2, the location points to the LanceDB database directory/prefix containing the .lance tables, rather than to an individual .lance table.

recursivezero · 2026-08-23T11:03:07Z

No table get scanned hence no result; tried with empty value also

recursivezero · 2026-08-23T11:10:59Z

for Local its working fine

recursivezero · 2026-08-23T11:11:31Z

Also create a separate markdown file for lancedb explorer not just under PR; how to make it work with screenshots

DhwaniKhandelwal-tech and others added 9 commits July 29, 2026 12:53

added admin dashboard

c92b78f

fixed lint issues in admin page

36099e8

Add default admin password placeholder

ecb795c

[TZF-260079] Fix LanceDB backend compatibility

e104d59

[TZF-260079] Complete LanceDB explorer frontend setup

00943f3

[TZF-260079] Back-merge TZF-260074 bugfix changes

75adc72

chore: update backend requirements.txt [skip ci]

27a78d3

[TZF-260079] Merge develop into LanceDB explorer feature

9dc8e49

Merge branch 'feature/TZF-260079' of https://github.com/recursivezero…

a765bf2

…/tz-fabric into feature/TZF-260079

DhwaniKhandelwal-tech requested a review from recursivezero as a code owner July 29, 2026 16:51

recursivezero added this to Threadzip Jul 29, 2026

github-project-automation Bot moved this to Todo in Threadzip Jul 29, 2026

recursivezero moved this from Todo to In Progress in Threadzip Jul 29, 2026

Admin Page Bug Fix

89e2db7

Add local and S3 LanceDB source selection

f49dd14

DhwaniKhandelwal-tech added 2 commits August 18, 2026 20:11

complete LanceDB scanner and S3 explorer changes

78be9f8

fixed lint issues

bf3f4a7

recursivezero reviewed Aug 20, 2026

View reviewed changes

fix: add R2 support, local table scanning, and LanceDB explorer valid…

4a4c3c9

…ation

DhwaniKhandelwal-tech added 2 commits August 21, 2026 19:48

fixed required changes

d8a56dd

Fixed the config issues

2e370ec

docs: add LanceDBexplorer setup and usage guide

f31d6b4

Navigation Menu

Uh oh!

[TZF-260079] Add admin-only LanceDB explorer - #82

[TZF-260079] Add admin-only LanceDB explorer#82
DhwaniKhandelwal-tech wants to merge 17 commits into
developfrom
feature/TZF-260079

DhwaniKhandelwal-tech commented Jul 29, 2026

Uh oh!

DhwaniKhandelwal-tech commented Jul 29, 2026

Uh oh!

recursivezero commented Jul 30, 2026 •

edited

Loading

Uh oh!

recursivezero commented Jul 30, 2026

Uh oh!

recursivezero commented Jul 30, 2026

Uh oh!

recursivezero commented Aug 1, 2026

Uh oh!

recursivezero commented Aug 4, 2026

Uh oh!

recursivezero Aug 4, 2026

Uh oh!

DhwaniKhandelwal-tech commented Aug 21, 2026

Used to authenticate the LanceDB Admin Explorer.

Choose a secure value for the environment.

Uh oh!

recursivezero commented Aug 23, 2026 •

edited

Loading

Uh oh!

recursivezero commented Aug 23, 2026

Uh oh!

recursivezero commented Aug 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

DhwaniKhandelwal-tech commented Jul 29, 2026

Summary

Screenshots

Administrator access gate

LanceDB table overview

Row and vector details

Backend changes

Frontend changes

Security

Validation

Backend

Frontend

Merge target

Uh oh!

DhwaniKhandelwal-tech commented Jul 29, 2026

Uh oh!

recursivezero commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

recursivezero commented Jul 30, 2026

Uh oh!

recursivezero commented Jul 30, 2026

Uh oh!

recursivezero commented Aug 1, 2026

Uh oh!

recursivezero commented Aug 4, 2026

Uh oh!

recursivezero Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

DhwaniKhandelwal-tech commented Aug 21, 2026

LanceDB Admin Explorer — Setup & Usage

1. Backend Environment Setup

Common configuration

Used to authenticate the LanceDB Admin Explorer.

Choose a secure value for the environment.

2. Local LanceDB Configuration

Important

3. Amazon S3 Setup

S3 — Database at bucket root

S3 — Database inside a prefix/folder

4. Cloudflare R2 Setup

Important distinction

5. R2 — Database at bucket root

6. R2 — Database inside a prefix/folder

7. Why R2 Uses an s3:// Location

8. Install and Run the Backend

9. Install and Run the Frontend

10. Login to LanceDB Explorer

11. Using the Explorer

Local

Amazon S3

Cloudflare R2

12. Table Selection

13. Available Explorer Functionality

14. Credentials & Security

Quick Reference

Example

Uh oh!

recursivezero commented Aug 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

recursivezero commented Aug 23, 2026

Uh oh!

recursivezero commented Aug 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

recursivezero commented Jul 30, 2026 •

edited

Loading

7. Why R2 Uses an `s3://` Location

recursivezero commented Aug 23, 2026 •

edited

Loading