High-performance LLM query cache with semantic search. Reduce API costs 80% and latency from 8.5s to 1ms using Redis + Qdrant vector DB. Multi-provider support (OpenAI, Anthropic).
Tools, libraries, papers, and patterns for reducing the cost of running large language models in production.
Self-hosted LLM control layer for cost optimization, policy enforcement, observability, and governance. OpenAI-compatible, with observe/enforce modes and optional security, privacy, and compliance modules.
Adaptive sliding window query debouncing and micro-batch coalescing engine optimizing LLM and tool call throughput
Adaptive sliding window query debouncing and micro-batch coalescing engine optimizing LLM and tool call throughput
Biological nervous systems don't recompute known workflows from scratch. Mnemon gives LLM agents the same primitive — execution memory that caches plans, not responses. 93% token reduction, 2.66ms vs 20s, zero tokens on repeat runs. LangChain, CrewAI, AutoGen.
Code and experimental artifacts for "Synchronous Online Verification Gating in Semantic Caches" — an empirical study of real-time verifier gating for semantic cache hits, evaluated against static/adaptive-threshold baselines on ~210k real requests across three datasets.
Verification layer for semantic caches: a REST API and Python client that checks gray-zone cache hits with a cross-encoder before they are served.
Self-hosted SemanticGuard deployment modules — bring the AI gateway with self-validating semantic caching into your own GCP / AWS / Azure account. https://www.semanticguard.dev
Cost-aware KV cache lease optimization + nonblocking long-task runtime for Hermes Agent (cache intelligence, provider-aware request replay)
High-performance semantic caching microservice for LLM queries powered by FastAPI, FastEmbed (bge-small-en-v1.5), and Redis Vector Search (VSS). Sub-10ms cache hit response times.
Read once. Brief anywhere. Reusable document briefings from fingerprinted cache.
High-performance Reverse Proxy for LLM Caching and Deduplication (OpenAI, Anthropic, Gemini, Copilot). Saves API costs, encrypts cache at rest, and multicasts concurrent streams.
Production-ready Retrieval-Augmented Generation (RAG) system using LangChain and Google Gemini. Features hybrid search (Chroma + BM25), reciprocal rank fusion, idempotent document ingestion, persistent SQLite LLM caching, an interactive CLI shell, and a Streamlit UI.
EDU AI LAB: Local AI is not CHEAP at ALL! - Every AI token carries a COST! How to measure them?
System 0 for your LLM: learns the fixed-choice answers your LLM gives (intents, routing, moderation) and serves the repeats in microseconds from a small CPU table, at most 5% disagreement, certified. Zero-code proxy for the OpenAI Decisions API, System One and chat models.
FreCoS: freshness- and cost-aware extension to GPTCache (BSc course project)
Pre-indexed code knowledge graph for Claude Code, Codex, Gemini, Cursor, OpenCode, AntiGravity, Kiro, and Hermes Agent — fewer tokens, fewer tool calls, 100% local
To associate your repository with the llm-caching topic, visit your repo's landing page and select "manage topics."