🚀 DSpark

Agent-Level Speculative Orchestration & Formal Dual-Engine Code Generation

📑 Research Paper • 📚 Documentation • 🚀 Quick Start • 📊 Benchmarks • 🔌 MCP IDE Setup • 🤝 Contributing

---

🎯 What is DSpark?

DSpark is an enterprise-grade AI coding platform and MCP server that elevates Speculative Decoding to the Agent Orchestration Level. It replaces expensive, brute-force model prompting with an efficient multi-tier architecture combining:

1. ⚡ Semi-Autoregressive Speculative Drafting: Generates $N$ parallel code candidates using high-speed/local models bounded by asynchronous semaphores.
2. 🌲 Sequential AST Dependency Resolution: Validates code syntax and topologically sorts function call graphs via Tree-Sitter/Regex before calling remote verification.
3. 🔍 Probabilistic Pivot Tournament (PPT): Evaluates candidates in $O(Nk)$ comparisons instead of naive all-pairs $O(N^2)$ using fine-grained reward estimation.
4. 📊 Confidence-Scheduled Pruning: Analyzes cyclomatic complexity and state mutations locally on CPU, pruning 60–98% of redundant API calls without compromising safety.
5. 🧠 Dual-Engine CEGAR Refinement: Epistemically isolates the Creator from the Curator (DeepSeek v4 Pro / Flash) with real sandbox execution and deterministic counterexamples (failure_tail).
6. 🔌 Universal MCP Server: Integrates natively into Cursor, Claude Code, Claude Desktop, Antigravity, Windsurf, and Roo Code.

Theoretical Foundations: Synthesized from DSpark (DeepSeek & Peking University, 2026) and LLM-as-a-Verifier (Kwok et al., 2026).

---

🏗️ Architecture Workflow

flowchart TD
    UserSpec["📋 User Spec + I/O Contracts"] --> Drafter["⚡ Stage 1: Speculative Drafter\n(N=3..5 parallel trajectories)"]
    Drafter --> AST["🌲 Stage 2: AST Dependency Resolver\n(Topological DAG Sort & Cycle Detection)"]
    AST --> ConfHead["📊 Stage 3: Local Confidence Head\n(CPU Entropy & Risk Assessment)"]
    
    ConfHead -->|"Low Risk (Pruned 60-98%)"| LocalApprove["✅ Local Zero-Cost Approval"]
    ConfHead -->|"High Risk / Ambiguity"| Scheduler["💰 Stage 4: Cost-Aware Scheduler\n(Verification Budget Cap)"]
    
    Scheduler --> PPT["🏆 Stage 5: Probabilistic Pivot Tournament\n(O(Nk) Pairwise Verifications)"]
    PPT --> Winner["🥇 Selected Trajectory"]
    
    Winner --> Sandbox{"🧪 Sandbox Verification\n(Pytest / Cargo Contracts)"}
    Sandbox -->|"PASS"| Done["🎉 Verified Production Code"]
    Sandbox -->|"FAIL (Counterexample)"| Curator["🧠 CEGAR Refiner (DeepSeek Flagship)\n(Epistemic Isolation + 1-Shot Fix)"]
    Curator --> Sandbox

Loading

---

🔬 Live Empirical Benchmarks

All metrics below are regenerable directly via python bench/run_real_bench.py and asserted in CI.

1. Accuracy & Token Arbitrage (12 Real Complex Tasks)

Configuration	Drafting Tier	Refinement Tier	Zero-Shot Pass@1	DSpark Tiered Pass@1	Total Spend
Weak Model Alone	gpt-3.5-turbo	None	41.7%	41.7%	$0.0035
DSpark Tiered Hybrid	gpt-3.5-turbo	deepseek-chat	41.7%	75.0% (+33.3 pts)	$0.0271
Flagship Standalone	deepseek-chat	None (1-shot)	91.7%	91.7%	$0.0050
DSpark Flagship Speculative	deepseek-chat	deepseek-chat	91.7%	100.0% (Perfect Score)	$0.0239

2. Token & Compute Reduction Breakdown

Layer / Mechanism	Baseline Approach	DSpark Speculative Approach	Token & Call Reduction
Flagship Token Offloading	100% tokens sent to Flagship	89.2% tokens handled by cheap/local tier	89.2% flagship tokens saved ✅
Tournament Comparisons ($N=100$)	4,950 all-pairs evaluations	394 PPT ring & anchor evaluations	92.0% comparison calls saved ✅
Local Risk & Entropy Pruning	Send all code blocks to remote API	CPU evaluates entropy & prunes trivial blocks	60.0%–98.0% API calls eliminated ✅
KV Prefix-Cache Optimization	Unordered dynamic prompt context	Invariant static contract prefix ordering	Up to 80.0% input token discount ✅

3. Probabilistic Pivot Tournament (PPT) Scaling ($O(Nk)$ vs $O(N^2)$)

Asserted over the wire by tests/tournament_scaling_test.rs:

Candidates ($N$)	Effective Pivots ($k$)	Tournament Comparisons	All-Pairs $O(N^2)$	Comparison Reduction
$N = 10$	3	34	45	24.4%
$N = 20$	3	74	190	61.1%
$N = 50$	3	194	1,225	84.2%
$N = 100$	3	394	4,950	92.0%

---

🔌 IDE & MCP Integration

DSpark includes a high-performance FastMCP Server exposing formal verification and speculative code generation to any AI-assisted editor.

1. Cursor & Windsurf (~/.cursor/mcp.json or mcp.json)

{
  "mcpServers": {
    "dspark": {
      "command": "python",
      "args": ["-m", "dspark.mcp.server"],
      "cwd": "C:/Users/adeil/dspark",
      "env": {
        "DEEPSEEK_API_KEY": "your-deepseek-key",
        "OPENAI_API_KEY": "your-openai-key"
      }
    }
  }
}

2. Claude Desktop & Claude Code (claude_desktop_config.json)

{
  "mcpServers": {
    "dspark-dual-engine": {
      "command": "python",
      "args": ["-m", "dspark.mcp.server"],
      "cwd": "C:/Users/adeil/dspark",
      "env": {
        "DEEPSEEK_API_KEY": "your-deepseek-key"
      }
    }
  }
}

3. Exposed MCP Tools

• dspark_audit: Formally audits code against AST-inferred or user-provided I/O contracts in an isolated sandbox.
• dspark_refine: Repairs failing code using epistemic isolation guided by concrete failing tracebacks (failure_tail).
• dspark_verify_pipeline: Executes the full speculative multi-trajectory CEGAR loop end-to-end.

---

🚀 Quick Start

Prerequisites

• Rust toolchain (1.75+): curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
• Python (3.10+): python --version
• API Keys: DeepSeek API Key, OpenAI API Key, or Gemini API Key.

Installation

# Clone the repository
git clone https://github.com/CostaJr007/dspark.git
cd dspark

# Install the Rust CLI (Fast regex AST backend)
cargo install --path crates/dspark-core --force

# OR install with Tree-Sitter AST feature
cargo install --path crates/dspark-core --features tree-sitter-ast --force

# Install the Python SDK & CLI
pip install -e .

Environment Configuration

# Linux / macOS
export DEEPSEEK_API_KEY="sk-..."
export OPENAI_API_KEY="sk-..."

# Windows PowerShell
$env:DEEPSEEK_API_KEY="sk-..."
$env:OPENAI_API_KEY="sk-..."

---

💻 CLI Usage Examples

1. Speculative Multi-Trajectory Generation

# Generate code with 4 parallel trajectories and 2 tournament pivots
dspark run "Implement a thread-safe LRU Cache with TTL expiration in Python" \
           --speculative \
           --trajectories 4 \
           --pivots 2 \
           --out lru_cache.py

2. Audit Code Against Formal Contracts

dspark audit path/to/module.py

3. Surgical Refinement via Counterexamples

dspark refine path/to/failing_code.py

4. Interactive Terminal Agent

dspark

---

🐍 Python SDK Example

import asyncio
from dspark.pipeline.cegar import CEGARPipeline

async def main():
    pipeline = CEGARPipeline()
    result = await pipeline.run(
        task_description="Implement a Trie autocomplete data structure with frequency ranking"
    )
    print(f"Status: {result.status}")
    print(f"Verified Code:\n{result.final_code}")

if __name__ == "__main__":
    asyncio.run(main())

---

🦀 Rust Crate Example

use dspark::client::ModelClient;
use dspark::engine::PivotTournament;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = ModelClient::from_spec("deepseek-v4-flash")?;
    let tournament = PivotTournament::new(client, 2);
    
    // Execute O(Nk) tournament ranking across draft candidates
    // let result = tournament.run_tournament(&trajectories, "Check correctness").await;
    Ok(())
}

---

📖 Documentation Index

Guide	Description
📑 Research Paper	"Beyond Passive Selection: Agent-Level Speculative Orchestration and CEGAR Refinement" (Preprint)
🏛️ Architecture	In-depth engineering specifications of the 5-stage pipeline and CEGAR loop
🚀 Getting Started	Step-by-step setup, configuration, and IDE integration guide
📊 Benchmarks & Methodology	Criterion scaling benchmarks, pilot results, and token economics
🎓 Theoretical Foundations	Academic foundations (DSpark, CEGAR, LLM-as-a-Verifier)
🔌 API Reference	Full Rust crate and Python SDK API reference
⌨️ CLI Reference	Complete CLI arguments, options, and commands
🤝 Contributing	Contribution guidelines, code standards, and PR workflows
📜 Changelog	Version history and milestone releases

---

🧪 Testing Suite

# Run all Rust tests (37 tests)
cargo test -p dspark-core

# Run all Python tests (16 tests)
pytest -v

---

📜 Attribution & Academic Lineage

Note

Academic Lineage & Inspiration:

• DeepSeek DSpark (2026): Inspired by the seminal work "DSpark: Confidence-Scheduled Speculative Decoding for Large Language Models" (DeepSeek-AI & Peking University, 2026), which pioneered confidence scheduling for speculative token generation on GPU runtimes. DSpark Agent abstracts and elevates these principles from token-level tensor scheduling to macro-level multi-agent software orchestration, AST dependency resolution, and CEGAR verification loops.
• LLM-as-a-Verifier (2026): Incorporates and extends the Probabilistic Pivot Tournament (PPT) algorithm formulated by Kwok et al. (2026), replacing passive candidate selection with deterministic sandbox repair.
• CLI Scaffolding: Builds upon and evolves open-source terminal agent scaffolding paradigms into a high-performance native Rust core (dspark-core) and FastMCP server.

---

📜 License

Distributed under the MIT License. See LICENSE for details.