feat(sdk): Evaluator v2 outbound worker runtime — managed + self-hosted pods, fork+exec sandbox, review hardening - #758

Open

SiddarthAA wants to merge 23 commits into

mainfrom

evaluator

Commits on Sep 3, 2026

docs(sdk): clarify evaluator v2 boundary

SiddarthAA
authored and
chhhee10
committed
4620e56
feat(sdk): add evaluator v2 worker runtime
chhhee10
committed
6642a0d
fix(evaluator): resume only unfinished replayed runs
chhhee10
committed
20e9c1e
feat(sdk): run hosted evaluator definitions
chhhee10
committed
30f2fc2
fix(evaluator): harden managed source sandbox and contain poison definitions

chhhee10
and
claude
committed
17bf0a6
feat(evaluator): normal short polling instead of long-poll in the worker

chhhee10
and
claude
committed
ca3a7c8
fix(evaluator): sandbox managed source in a killable process (hermes SEC-001)

chhhee10
and
claude
committed
6a3592b
fix(evaluator): fail closed when the fork sandbox is unavailable (hermes SEC-001)

chhhee10
and
claude
committed
e955eef
fix(evaluator): fork+exec the sandbox + clamp the timeout (hermes SEC-001)

chhhee10
and
claude
committed
dadd90f
fix(evaluator): bound the sandbox result on both sides (hermes SEC-001)

chhhee10
and
claude
committed
6a7aea8
fix(evaluator): bound aggregate sandbox memory (hermes SEC-001)

chhhee10
and
claude
committed
ff95a42
Harden evaluator v2 sandbox; fix condition selection and error mirror

chhhee10
and
claude
committed
0868e00
Count sandbox-slot wait against the execution timeout (hermes SEC-001)

chhhee10
and
claude
committed
c1e63ca
fix(evaluator): report why source was rejected; scrub the sandbox child's env

authored and
chhhee10
committed
05b6266
test(evaluator): give the condition compute bomb a margin that survives a fast interpreter

authored and
chhhee10
committed
b24e8c3
fp-cloud-cli: mirror the new evaluations:run permission

chhhee10
and
claude
committed
e5a29d7
sdk(evaluator): fail closed when kernel resource limits are unavailable

chhhee10
and
claude
committed
e435035
sdk(evaluator): bound the condition phase by the lease; make sync-eval thread leaks observable

chhhee10
and
claude
committed
47cd2c8
sdk(evaluator): let a managed comprehension read session on Python 3.10 (hermes COR-001)

chhhee10
and
claude
committed
b3dd423
chore(security): time-box the nltk advisory that has no fix

chhhee10
and
claude
committed
f6770f2

Commits on Sep 4, 2026

sdk(evaluator): fix five worker-runtime defects found in review

SiddarthAA
and
claude
committed
6e0fdec
sdk(evaluator): register insufficient_permissions in the error contract

SiddarthAA
and
claude
committed
3450052
sdk(evaluator): harden the run loop against lease loss, replay, and too-large transcripts

SiddarthAA
and
claude
committed
ecb3e08

Navigation Menu

Uh oh!

feat(sdk): Evaluator v2 outbound worker runtime — managed + self-hosted pods, fork+exec sandbox, review hardening - #758

feat(sdk): Evaluator v2 outbound worker runtime — managed + self-hosted pods, fork+exec sandbox, review hardening#758
SiddarthAA wants to merge 23 commits into
mainfrom
evaluator

Commits on Sep 3, 2026

docs(sdk): clarify evaluator v2 boundary

feat(sdk): add evaluator v2 worker runtime

fix(evaluator): resume only unfinished replayed runs

feat(sdk): run hosted evaluator definitions

fix(evaluator): harden managed source sandbox and contain poison definitions

feat(evaluator): normal short polling instead of long-poll in the worker

fix(evaluator): sandbox managed source in a killable process (hermes SEC-001)

fix(evaluator): fail closed when the fork sandbox is unavailable (hermes SEC-001)

fix(evaluator): fork+exec the sandbox + clamp the timeout (hermes SEC-001)

fix(evaluator): bound the sandbox result on both sides (hermes SEC-001)

fix(evaluator): bound aggregate sandbox memory (hermes SEC-001)

Harden evaluator v2 sandbox; fix condition selection and error mirror

Count sandbox-slot wait against the execution timeout (hermes SEC-001)

fix(evaluator): report why source was rejected; scrub the sandbox child's env

test(evaluator): give the condition compute bomb a margin that survives a fast interpreter

fp-cloud-cli: mirror the new evaluations:run permission

sdk(evaluator): fail closed when kernel resource limits are unavailable

sdk(evaluator): bound the condition phase by the lease; make sync-eval thread leaks observable

sdk(evaluator): let a managed comprehension read session on Python 3.10 (hermes COR-001)

chore(security): time-box the nltk advisory that has no fix

Commits on Sep 4, 2026

sdk(evaluator): fix five worker-runtime defects found in review

sdk(evaluator): register insufficient_permissions in the error contract

sdk(evaluator): harden the run loop against lease loss, replay, and too-large transcripts