A production-grade, distributed hybrid search engine in Go, purpose-built for retrieval-augmented generation (RAG) and agent-native search. YASE fuses lexical and semantic retrieval — BM25 full-text search (Bluge) and HNSW vector similarity — into a single pipeline using Reciprocal Rank Fusion and an optional cross-encoder reranker, then serves results as citable, attributed evidence for LLMs.
- Hybrid search — BM25 + HNSW-IF vector search, fused via Reciprocal Rank Fusion with an optional cross-encoder reranker.
- Pre-filtering, not post-filtering — metadata and tenant filters run before vector math via Bluge Roaring Bitmaps, so restricted users never see documents they can't access (a common failure mode of pure vector databases).
- Agentic RAG API —
/v1/ragreturns inline[N]citations, source URLs, chunk text, and per-result scores, ready to inject straight into an LLM prompt. - Semantic + AST-aware chunking — cosine-similarity valley detection groups prose by topic; tree-sitter keeps code units (functions, classes) intact.
- Local, self-hosted embeddings — TEI with GTE-ModernBERT-base (768-dim, MTEB 64.38) keeps data in-house at zero per-token cost. Ollama and OpenAI providers included.
- Collections — physically isolated search spaces with per-collection embedders and config, plus cross-collection fan-out merged via RRF.
- 10 enterprise connectors — Confluence, Jira, S3, PostgreSQL, MySQL, Google Drive, Salesforce, SharePoint, Slack, Twitter/X, plus a go-plugin system for custom sources.
- Distributed scale-out — Raft consensus, consistent hashing, and scatter-gather across shards.
- Production-ready — API-key/JWT auth, TLS, tenant isolation, rate limiting, audit logging, JSON logging, OpenTelemetry tracing, Prometheus metrics, and a Helm chart.
Retrieval for LLM agents differs from classic search: agents need correctly-scoped, citable evidence with full provenance — not just a ranked list of links. YASE is built around that contract.
| Agent requirement | How YASE delivers it |
|---|---|
| Grounded answers | /v1/rag returns citations with doc_id, source_url, chunk_text, and [N] markers to pin claims to evidence |
| Correct access control | _tenant filters are injected server-side and cannot be overridden, scoping every query to the caller's data |
| Coherent context windows | Semantic valley + AST-aware chunking keep topics and functions whole — no mid-thought splits |
| Keyword + semantic precision | 5-stage pipeline: BM25 pre-filter → HNSW traversal → cosine rescore → RRF → cross-encoder rerank |
| Whole-corpus recall | Cross-collection fan-out searches every collection and merges with RRF — no global "superset" duplication |
| Data sovereignty | Self-hosted TEI embeddings — no documents or queries leave your infrastructure |
| Fresh indexes | Deterministic chunk IDs + delete-before-write mean updated sources never leave stale chunks |
| Full provenance | Every result carries its metadata (URL, author, timestamps) for transparent attribution |
One call, ready for your LLM:
curl -s -X POST http://localhost:8000/v1/rag \
-H "Content-Type: application/json" \
-d '{"query": "How does HNSW work?", "top_k": 5, "include_text": true}' | jq{
"status": "success",
"query": "How does HNSW work?",
"citations": [
{
"doc_id": 12345,
"chunk_text": "HNSW builds a multi-layer graph where top layers provide long-range shortcuts...",
"source_url": "https://example.com/hnsw.md",
"relevance_score": 0.0421,
"bm25_score": 8.2,
"semantic_score": 0.87,
"metadata": { "source": "confluence", "space": "ENG" }
}
],
"context": "[1] HNSW builds a multi-layer graph where...\n\n[2] Reciprocal Rank Fusion merges...",
"duration_ms": 42
}The assembled context string with [1], [2], … markers can be injected directly into a system or user prompt for grounded generation.
┌─────────────────┐ ┌──────────────────┐ ┌──────────────────────┐
│ Connectors │ │ Crawler │ │ Direct Ingest │
│ Confluence/Jira │ │ Master/Worker │ │ gRPC :50051 │
│ S3/Postgres/... │ │ │ │ │
└────────┬────────┘ └────────┬─────────┘ └──────────┬───────────┘
│ _collection_id │ │
└──────────────┬───────┘──────────────────────────┘
│ Kafka (crawl-records)
▼
┌─────────────────────┐
│ Indexer │ ┌───────────────┐
│ Collection Manager │────►│ TEI Embedder │
│ routes by │ │ :8888 │
│ _collection_id │ └───────────────┘
│ ┌───────┐┌───────┐│
│ │_default││col-A ││ ← each collection has own
│ │ Engine ││Engine ││ Bluge + HNSW-IF + Arena
│ └───────┘└───────┘│
└─────────┬───────────┘
│ gRPC
┌─────────▼───────────┐ ┌───────────────┐
│ Gateway │────►│ TEI Reranker │
│ :8000 (HTTP) │ │ :8081 │
│ /search │ └───────────────┘
│ /v1/collections/* │
└─────────────────────┘
# 1. Start infrastructure
docker compose -f deploy/docker-compose.yml up -d redis kafka
# 2. Start TEI embedder (builds from source on first run for Apple Silicon)
~/bin/text-embeddings-router --model-id Alibaba-NLP/gte-modernbert-base --port 8888 &
# 3. Start the combined indexer + gateway
YASE_EMBEDDER_PROVIDER=tei go run cmd/local/main.go &
# 4. Start ingestion
go run cmd/ingestion/main.go &
# 5. Search
curl -s -X POST http://localhost:8000/search \
-H "Content-Type: application/json" \
-d '{"query": "Go programming language", "top_k": 5}' | jqSee Getting Started for the full walkthrough (RAG, autocomplete, delete, crawler, connectors).
| Doc | Description |
|---|---|
| Getting Started | Full local setup, RAG, autocomplete, delete, crawler, connectors |
| Architecture | Services, search pipeline, project structure |
| Collections | Data isolation, cross-collection search, management |
| Connectors & Embeddings | Enterprise data sources and embedding providers |
| Configuration & Security | Config reference, auth, TLS, tenant isolation, observability |
| API Reference | Endpoints + Go client SDK |
| Deployment & Operations | Docker Compose, Helm, build & test |
| End-to-End Testing | Local 3-node distributed E2E: infra setup, colima, TEI, defects found |
Apache 2.0