From 0b95fb23a19a578f8fb5c5fa100fb033f9805551 Mon Sep 17 00:00:00 2001 From: Ragnor Comerford Date: Tue, 20 Oct 2026 17:47:58 -0700 Subject: [PATCH] docs: state encryption, storage qualification and embedding egress The tree said nothing about encryption at rest, claimed R2 and GCS support that CI never exercises, and did not say when an embedding provider is called. A new data-protection page covers encryption, transport, egress and erasure; the storage page names the conditional writes a store must enforce; the embeddings page says which operations reach a provider and that no setting turns providers off. --- README.md | 10 +++--- docs/user/concepts/storage.md | 9 +++++ docs/user/deployment.md | 8 ++--- docs/user/index.md | 1 + docs/user/operations/data-protection.md | 44 +++++++++++++++++++++++++ docs/user/search/embeddings.md | 22 +++++++++++++ 6 files changed, 86 insertions(+), 8 deletions(-) create mode 100644 docs/user/operations/data-protection.md diff --git a/README.md b/README.md index 39cd7dd6f..16eee31d1 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ Omnigraph is engineered for the new workload introduced by long-horizon agents: | **Branching** | Agents propose reviewable changes instead of writing to main: a branch per agent or task, three-way merge, time travel across the whole graph. | | **Declared as code** | A `cluster.yaml` declares graphs, schemas, stored queries, embedding providers and policies. `cluster plan` previews, `cluster apply` converges, Terraform-style. | | **Security as code** | Cedar policy enforced **server-side on every mutation**, per graph or server-wide; bearer auth, server-resolved actors, audit tracking. | -| **Object-storage native, open format** | Scales at the lowest cost on local disk or any S3-compatible store: RustFS / MinIO, AWS S3 / R2 / GCS, Azure preview. Data lives in open, versioned [Lance](https://github.com/lance-format/lance) with native blob-as-data, and never leaves your store. | +| **Object-storage native, open format** | Scales at the lowest cost on local disk, AWS S3 or a self-hosted S3-compatible store such as RustFS, with Azure in preview. Data lives in open, versioned [Lance](https://github.com/lance-format/lance) with native blob-as-data, and never leaves your store. | ## What you can build @@ -137,9 +137,11 @@ policies: **2. Stand up your object store.** On-prem, run RustFS (or MinIO); Omnigraph writes [Lance](https://github.com/lance-format/lance) over the standard S3 API. -In the cloud, use S3 / R2 / GCS through `AWS_*`. Native `az://` roots are also -available as a qualification preview. Every Azure writer, including -`cluster apply` and the server, must use the checked-in admission wrapper; +In the cloud, use AWS S3 through `AWS_*`; other S3-compatible services must +support [conditional writes](docs/user/concepts/storage.md#supported-graph-uris). +Native `az://` roots are also available as a qualification preview. Every +Azure writer, including `cluster apply` and the server, must use the +checked-in admission wrapper; follow the [Azure deployment guide](docs/user/deployment.md#azure-blob-preview) instead of running the bare commands below. diff --git a/docs/user/concepts/storage.md b/docs/user/concepts/storage.md index cd2732775..8fb36464e 100644 --- a/docs/user/concepts/storage.md +++ b/docs/user/concepts/storage.md @@ -13,6 +13,15 @@ datasets directly. | `s3://bucket/prefix/graph.omni` | Amazon S3 or an S3-compatible service | | `az://container/prefix/graph.omni` | Azure Blob Storage qualification preview | +Other schemes, including `gs://`, are refused; there is no native Google Cloud +Storage support. Writers rely on conditional writes: the store must refuse to +create an object that already exists (`If-None-Match: *`) and refuse to replace +one whose ETag has changed (`If-Match`). Amazon S3 supports both, and CI tests +the `s3://` path against RustFS. Other S3-compatible services, such as +Cloudflare R2 or Google Cloud Storage through its S3-compatible API, are not +qualified. Confirm that a service enforces both conditions before you write to +it: one that ignores them can let two writers overwrite each other's commits. + An `http://` or `https://` URL addresses an OmniGraph server; it is not a graph storage URI. CLI commands use `--target NAME|ADDRESS` for both server and direct-storage destinations; the selected kind determines command availability. diff --git a/docs/user/deployment.md b/docs/user/deployment.md index e50a21d87..e06365c08 100644 --- a/docs/user/deployment.md +++ b/docs/user/deployment.md @@ -210,7 +210,8 @@ docker run --rm -p 8080:8080 \ ``` Terminate TLS at a load balancer or trusted reverse proxy. Keep bearer tokens -and storage credentials in the platform's secret store. +and storage credentials in the platform's secret store. For encryption at rest +and data sent to external services, see [data protection](operations/data-protection.md). ## Unix-socket access @@ -331,9 +332,8 @@ Configure tokens from environment or a mounted secret file: - `OMNIGRAPH_SERVER_BEARER_TOKENS_FILE` - `OMNIGRAPH_SERVER_BEARER_TOKENS_AWS_SECRET` in the AWS-enabled build -Policy bundles come from the applied cluster. See -[HTTP server](operations/server.md) and -[Authorization and actors](operations/policy.md). +Policy bundles come from the applied cluster. See [HTTP server](operations/server.md) +and [Authorization and actors](operations/policy.md). ## Upgrade and backup diff --git a/docs/user/index.md b/docs/user/index.md index f5c22a35c..a00157d6b 100644 --- a/docs/user/index.md +++ b/docs/user/index.md @@ -46,6 +46,7 @@ deployment. - [Deployment](deployment.md) - [Storage failures](operations/storage-failures.md) - [Authorization and actors](operations/policy.md) +- [Data protection](operations/data-protection.md) - [Maintenance](operations/maintenance.md) - [Storage-format upgrades](operations/upgrade.md) diff --git a/docs/user/operations/data-protection.md b/docs/user/operations/data-protection.md new file mode 100644 index 000000000..d68e725f3 --- /dev/null +++ b/docs/user/operations/data-protection.md @@ -0,0 +1,44 @@ +# Data protection + +OmniGraph does not encrypt what it writes. It has no client-side or field-level +encryption: graph data, history, managed Blob bytes, schemas, stored queries and +policy bundles are stored in the cluster root as written. Protecting them at +rest and in transit is the job of the storage service and the network in front +of the server. + +## Encryption at rest + +Use the storage service's encryption: + +- **Amazon S3.** Default encryption on the bucket (SSE-S3 or SSE-KMS) applies + to every object OmniGraph writes, with no OmniGraph setting. With a + customer-managed KMS key, the credentials OmniGraph uses also need permission + to use that key. Per-request settings such as `AWS_SERVER_SIDE_ENCRYPTION` + are not needed, and CLI [targets](../cli/targets.md) refuse them. +- **Azure Blob Storage.** Azure Storage encrypts Blob data at rest by default. +- **Local filesystem.** Put the root on an encrypted volume. + +## Encryption in transit + +`omnigraph-server` serves plain HTTP. Terminate TLS in front of it, as +described under [Container](../deployment.md#container). Storage traffic uses +HTTPS unless you configure an `http://` endpoint for local development. + +## Data sent to external services + +The only external service OmniGraph sends text to is the embedding provider. A +text `nearest` query sends its query text, and the offline `omnigraph embed` +command sends the values it embeds. Stored graph data is never sent. See +[network egress](../search/embeddings.md#network-egress) for when a provider is +called and how a graph selects one. + +## Erasure + +Deleting data or dropping a property changes the current graph only. Earlier +commits keep the old values readable until [cleanup](maintenance.md#cleanup) +removes them, and the bytes can remain in storage files after that. To erase a +dropped property's values, follow the procedure in +[schema changes](../schema/index.md#schema-changes). + +This is not secure erasure. Object-store versioning, retention policies and +backups can keep copies that OmniGraph does not manage. diff --git a/docs/user/search/embeddings.md b/docs/user/search/embeddings.md index 88954373b..b0987ecbc 100644 --- a/docs/user/search/embeddings.md +++ b/docs/user/search/embeddings.md @@ -57,6 +57,28 @@ graphs: Inline API keys are rejected. `${ENV_VAR}` references are resolved when the server starts, not when the cluster configuration is planned or applied. +## Network egress + +OmniGraph calls the embedding provider only to turn text into a vector: + +- a `nearest` query whose argument is text sends that text; +- `omnigraph embed` sends each source value it embeds. + +Loads, mutations and `nearest` queries given a vector never call a provider; +`@embed` does not populate vectors during a load +([schema annotation](#schema-annotation)). + +A cluster-served graph uses the provider its `embedding_provider` names. A +served graph without one, and every direct or embedded use, resolves the +provider from the process environment as described above; with nothing set, +that is OpenRouter. When the provider's API key is missing, the call fails +before any request is sent. + +No setting turns providers off. `mock` makes no network calls, but it answers +text queries with deterministic vectors that are not comparable with real +ones. A text `nearest` on a property whose `@embed` records a `model` is +refused under `mock` unless `OMNIGRAPH_EMBED_MODEL` names that model. + ## Schema annotation Associate a vector with its source text: