Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ ruff format --check .
pytest
```

Тесты используют fake/deterministic providers и не требуют внешних AI API. PostgreSQL/MCP integration-тесты требуют `DATABASE_URL` и выполняются в CI с pgvector.
Тесты используют fake/deterministic providers и не требуют внешних AI API. Test-only scripted HTTP endpoint дополнительно проверяет сквозной путь provider → model patch → validation → worktree → sandbox → repair/audit; его mock diff не считается генерацией модели и не участвует в benchmark. Такая симуляция подтверждает интеграцию и защитный контур, но не качество генерации, производительность, GPU, offline или работу настоящей модели. Embeddings пока fake. PostgreSQL/MCP integration-тесты требуют `DATABASE_URL` и выполняются в CI с pgvector.

## Связанные репозитории

Expand Down
1 change: 1 addition & 0 deletions docs/DEVELOPMENT_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@
- Реализовать сменный provider для OpenAI-compatible локального endpoint без привязки к inference server; fake HTTP server tests не обращаются к внешним AI API.
- Подключить настроенную LLM к document/code RAG; embeddings пока остаются fake и должны быть отражены в итоговых выводах.
- Подключить model-based patch generation к существующим `PatchProvider` и coding orchestration; валидировать tool calls по allowlist/schema и unified diff до `git apply`.
- Проверять сквозной fake HTTP → model patch → worktree → sandbox → repair/audit путь только scripted test fixture; это integration simulation, не model generation benchmark и не hardware validation.
- Запустить quantized coding model на RTX 4070 Ti: сначала 7B, затем сравнить с 14B при наличии VRAM.
- Проверить initial patch и максимум две repair iterations на синтетических coding tasks.
- Сравнить полный ограниченный source snapshot без RAG, LLM + grounded code RAG и LLM + RAG + test loop.
Expand Down
4 changes: 4 additions & 0 deletions docs/EVALUATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,10 @@ Grounded path дополнительно проверяет существова

На M7 `copilot-coding-evaluation` сравнивает три режима на одном commit и наборе coding-задач: Base (полный ограниченный snapshot tracked Python-файлов из Git, без retrieval и repair), RAG (grounded code citations, без repair) и RAG + test loop (grounded citations, максимум две repair iterations). Все три режима получают исходный код: Base получает snapshot до 24 KiB суммарно, RAG — только выбранные commit-matched citations. `evals/coding_benchmark.json` содержит только task ID и instructions; loader отвергает дополнительные поля и runner не загружает prepared patch files. `patch_applied` и `patch_apply_rate` относятся к итоговому worktree diff после последней попытки, а не к промежуточной попытке, которую repair сбросил. Cleanup error сохраняется в результате задачи и не прерывает отчёт. Отчёт содержит SHA-256 canonical task set, base commit, configured model/endpoint, rates применения patch/build/test, число repairs, длительность, p50/p95 LLM request latency и tokens/sec, если endpoint возвращает usage. `pass_at_1` измеряется для Base/RAG; `success_after_repairs` отдельно отражает RAG + test loop. Human acceptance, VRAM и RAM автоматически не предполагаются и остаются ручными измерениями. Набор мал и синтетический, его результаты не доказывают качество на реальном коде.

### Fake HTTP integration simulation

Test-only endpoint в `tests/fake_inference.py` возвращает scripted mock responses, включая явно обозначенный `MOCK_NOTIONAL_DIFF`. Сквозные тесты проверяют реальный OpenAI-compatible HTTP provider, `ModelPatchProvider`, diff validation, Git worktree, bounded repair и audit; отдельный Docker-тест завершает этот путь реальными compileall/unittest. Fixture не загружается benchmark runner и не засчитывается как model-generated patch. Успешная симуляция доказывает только совместимость компонентов и работу защитного контура. Она не измеряет качество генерации, VRAM/RAM, latency настоящей модели, GPU, offline или production-readiness. Embeddings остаются fake; hardware/model gate на RTX 4070 Ti обязателен отдельно.

### Производительность

- p50/p95 latency и time to first token;
Expand Down
2 changes: 1 addition & 1 deletion docs/RUNBOOK.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ python -m local_coding_copilot.evaluation

Evaluation сохраняет `data/runtime/m3-evaluation.json`, сравнивает baseline и grounded RAG на одном SHA-256 dataset и проверяет recall@5, correct refusal и false refusal rate. CI сохраняет отчёт как artifact для каждой версии Python.

Без `DATABASE_URL` PostgreSQL/MCP integration-тесты пропускаются. В CI они выполняются с реальным pgvector на Python 3.11 и 3.13, включая cross-project ACL, HTTP, MCP, rate-limit и audit-сценарии. Внешние AI API не используются.
Без `DATABASE_URL` PostgreSQL/MCP integration-тесты пропускаются. Без Docker daemon пропускаются sandbox integration-тесты. В CI эти проверки выполняются с реальными pgvector/Docker на Python 3.11 и 3.13, включая cross-project ACL, HTTP, MCP, rate-limit, audit и сквозной fake inference → model patch → sandbox сценарий. Scripted mock diff находится только в test fixtures, не загружается benchmark и не считается генерацией модели. Внешние AI API не используются; успешная симуляция не заменяет hardware/model validation на целевом ПК.

## Будущая локальная конфигурация

Expand Down
73 changes: 73 additions & 0 deletions tests/fake_inference.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
"""Scripted test-only inference responses; nothing here is generated by a model."""

# ruff: noqa: E501

import json
from collections.abc import Iterator, Sequence
from contextlib import contextmanager
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread

MOCK_NOTIONAL_DIFF = """diff --git a/src/exchange_demo/orders.py b/src/exchange_demo/orders.py
--- a/src/exchange_demo/orders.py
+++ b/src/exchange_demo/orders.py
@@ -16,4 +16,4 @@ def validate_order(order: Order) -> None:
if order.quantity <= 0:
raise ValueError("Quantity must be positive")
- if order.price * order.quantity >= MAX_ORDER_NOTIONAL:
+ if order.price * order.quantity > MAX_ORDER_NOTIONAL:
raise ValueError("Order notional exceeds the maximum")
diff --git a/tests/test_orders.py b/tests/test_orders.py
--- a/tests/test_orders.py
+++ b/tests/test_orders.py
@@ -7,6 +7,5 @@ class OrderValidationTest(unittest.TestCase):
def test_valid_order_is_accepted(self) -> None:
validate_order(Order("a", price=100, quantity=10))

- @unittest.expectedFailure
def test_notional_boundary_is_accepted(self) -> None:
validate_order(Order("boundary", price=10_000, quantity=100))
"""


@contextmanager
def fake_inference_endpoint(
responses: Sequence[str],
) -> Iterator[tuple[str, list[dict[str, object]]]]:
scripted_responses = iter(responses)
requests: list[dict[str, object]] = []

class Handler(BaseHTTPRequestHandler):
def do_POST(self) -> None:
payload = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
requests.append({"path": self.path, **payload})
content = next(scripted_responses)
body = json.dumps(
{
"choices": [
{
"finish_reason": "stop",
"message": {"role": "assistant", "content": content},
}
],
"usage": {"prompt_tokens": 10, "completion_tokens": 5},
}
).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)

def log_message(self, format: str, *args: object) -> None:
return

server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
thread = Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
yield f"http://127.0.0.1:{server.server_port}/v1", requests
finally:
server.shutdown()
thread.join(timeout=2)
server.server_close()
116 changes: 116 additions & 0 deletions tests/test_coding_http_integration.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
import subprocess
from collections.abc import Sequence
from pathlib import Path

from fake_inference import MOCK_NOTIONAL_DIFF, fake_inference_endpoint

from local_coding_copilot.auth import Principal
from local_coding_copilot.coding.models import CodingTask, SandboxResult
from local_coding_copilot.coding.orchestrator import CodingOrchestrator
from local_coding_copilot.coding.patches import ModelPatchProvider
from local_coding_copilot.coding.worktree import GitWorktreeManager
from local_coding_copilot.providers import OpenAICompatibleLLMProvider


class RecordingSandbox:
def __init__(self) -> None:
self.calls: list[tuple[str, Path]] = []

def run(self, worktree: Path, command: Sequence[str], stage: str) -> SandboxResult:
self.calls.append((stage, worktree))
return SandboxResult(
stage=stage,
command=tuple(command),
exit_code=0,
duration_ms=1,
timed_out=False,
stdout_tail="synthetic success",
stderr_tail="",
)


class RecordingAudit:
def __init__(self) -> None:
self.events: list[tuple[object, ...]] = []

def record_audit(self, *args: object, **kwargs: object) -> None:
self.events.append(args)


def test_fake_http_drives_model_patch_repair_worktree_sandbox_and_audit(
exchange_repository: Path, tmp_path: Path, admin_principal: Principal
) -> None:
sandbox = RecordingSandbox()
audit = RecordingAudit()
worktrees = GitWorktreeManager(tmp_path / "worktrees")
base_commit = worktrees.resolve_base_commit(exchange_repository)
original_source = (exchange_repository / "src/exchange_demo/orders.py").read_text()

with fake_inference_endpoint(
("this mock response is not a diff", MOCK_NOTIONAL_DIFF)
) as (endpoint, requests):
provider = OpenAICompatibleLLMProvider(endpoint, "test-only-mock-coder")
result = CodingOrchestrator(
worktrees=worktrees,
sandbox=sandbox,
patches=ModelPatchProvider(provider),
audit=audit,
).run(
exchange_repository,
CodingTask(
"notional-boundary",
"Accept the synthetic exact boundary",
context=("Synthetic source fixture",),
context_commit_sha=base_commit,
),
"default",
principal=admin_principal,
transport="test",
)

assert provider.stats.requests == 2
assert provider.stats.failures == 0

assert result.status == "awaiting_review"
assert result.repair_iterations == 1
assert result.patch_applied
assert result.tests_changed
assert result.changed_files == (
"src/exchange_demo/orders.py",
"tests/test_orders.py",
)
assert result.build is not None and result.build.exit_code == 0
assert result.test is not None and result.test.exit_code == 0
assert sandbox.calls == [
("build", Path(result.worktree)),
("test", Path(result.worktree)),
]
assert Path(result.worktree).is_dir()
assert (exchange_repository / "src/exchange_demo/orders.py").read_text() == (
original_source
)
assert (
subprocess.run(
["git", "-C", str(exchange_repository), "status", "--porcelain"],
check=True,
capture_output=True,
text=True,
).stdout
== ""
)

assert len(requests) == 2
assert all(request["path"] == "/v1/chat/completions" for request in requests)
assert all(request["model"] == "test-only-mock-coder" for request in requests)
repair_prompt = requests[1]["messages"][1]["content"]
assert "Attempt: 2" in repair_prompt
assert "Stage: patch_validation" in repair_prompt
assert "expected only raw unified diff" in repair_prompt

assert len(audit.events) == 1
event = audit.events[0]
assert event[4] == "awaiting_review"
assert event[-1]["repair_iterations"] == 1
assert event[-1]["diff_sha256"] == result.diff_sha256
assert MOCK_NOTIONAL_DIFF not in str(event)
assert "Accept the synthetic exact boundary" not in str(event)
31 changes: 30 additions & 1 deletion tests/test_coding_sandbox.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,15 @@
from pathlib import Path

import pytest
from fake_inference import MOCK_NOTIONAL_DIFF, fake_inference_endpoint

from local_coding_copilot.auth import Principal
from local_coding_copilot.coding.models import CodingTask
from local_coding_copilot.coding.orchestrator import CodingOrchestrator
from local_coding_copilot.coding.patches import StaticPatchProvider
from local_coding_copilot.coding.patches import ModelPatchProvider, StaticPatchProvider
from local_coding_copilot.coding.sandbox import DockerSandbox, SandboxConfig
from local_coding_copilot.coding.worktree import GitWorktreeManager
from local_coding_copilot.providers import OpenAICompatibleLLMProvider

PROJECT_ROOT = Path(__file__).resolve().parents[1]

Expand Down Expand Up @@ -158,6 +160,33 @@ def test_prepared_patch_passes_real_build_and_tests(
assert result.tests_changed


def test_fake_http_model_patch_passes_real_build_and_tests(
exchange_repository: Path, tmp_path: Path, admin_principal: Principal
) -> None:
with fake_inference_endpoint((MOCK_NOTIONAL_DIFF,)) as (endpoint, requests):
provider = OpenAICompatibleLLMProvider(endpoint, "test-only-mock-coder")
result = CodingOrchestrator(
worktrees=GitWorktreeManager(tmp_path / "worktrees"),
sandbox=DockerSandbox(SandboxConfig(timeout_seconds=30)),
patches=ModelPatchProvider(provider),
audit=RecordingAudit(),
).run(
exchange_repository,
CodingTask("notional-boundary", "Apply the synthetic mock fixture"),
"default",
principal=admin_principal,
)

assert result.status == "awaiting_review", result.message
assert result.repair_iterations == 0
assert result.build is not None and result.build.exit_code == 0
assert result.test is not None
assert result.test.exit_code == 0, result.test.stdout_tail + result.test.stderr_tail
assert result.patch_applied
assert result.tests_changed
assert len(requests) == 1


class RecordingAudit:
def record_audit(self, *args: object, **kwargs: object) -> None:
pass
Loading