[caiovicentino]

Ekbasis-Skopos-27B is the next version of Ekbasis-27B: Ekbasis-27B trained further on executed shell, git, SQLite,
PostgreSQL and Python scenarios, with the same prompt, readout, API and client. This pull request adds its release
evaluation, the code and generators that produced its training data, its pre-registrations, and the docs that introduce
it. The client package (ekbasis/) is not changed. The weights are on Hugging Face
(Ekbasis-Skopos-27B, FP8 and INT4 builds), the training and
evaluation rows in ekbasis-skopos-data.

What is added

• RELEASE_EVAL_SKOPOS.md: the release evaluation, written by scripts/write_release_eval_skopos.py from
  results_skopos/ (no number is typed in it).
• results_skopos/: every summary, every per-item prediction, the scripts that computed them as they ran, and
  scripts/verify_summaries.py, which recomputes the summaries from the per-item files; the overlap audit
  (overlap_audit/), the candidates that were not released (candidates/), and the certified-mode carry-over check
  (certified_carryover/).
• prereg_skopos/: the pre-registrations, with their sha256 in prereg_skopos/PREREG_sha256.txt.
• results_skopos/prelaunch/: the pre-launch evaluation (prereg_skopos/PRELAUNCH_EVAL.md): one comparison per item,
  the battery's code as it ran and its outputs, every reference file with its sha256 (references.tsv), how it ran
  (DEVIATIONS_RUN.md), and the owner's release decision (RELEASE_DECISION.md).
• training/skopos/: the two training stages, the interpolation, merges and quantization as run, the lineage script,
  and stage_files_skopos.py, which rebuilds every stage's training file from ekbasis-skopos-data, with the exceptions
  its README lists; a rerun retrains the models approximately (the README's "What will differ in a rerun").
• generators/: the 78 programs that computed the answers of every training row and evaluation set of the lineage, as
  they ran (generators/provenance.json); MANIFEST.in keeps them out of the source distribution.
• README.md, CHANGELOG.md, MODEL_CARD.md, docs/VERIFY.md, docs/REFERENCES.md, scorecard/README.md: the new
  model introduced, with numbers written from results_skopos/ by scripts/apply_skopos_docs.py; Ekbasis-27B's
  sections are unchanged.
• tests/test_safer.py: a timing margin in one test. test_hook_server_down_and_slow_search gave the hook 1 s and
  failed under CPU load when the original check (which runs git) needed longer, the hook then asking as "could not
  foresee", as designed; it now gives 5 s, with every slowed search request outlasting it. No other test changed.

Results (against Ekbasis-27B on the same items)

• Release read (PREREG_R4_RELEASE_READ.md), bf16, served: 8 of 9 criteria
  hold. The one that fails is ftest_family's 629 answers the actions change, −1.59 points (the limit is −1). A
  confirmation registered after that result (before any fresh item existed), by trainer readout
  (CONFIRM_FTEST.md), did not confirm the loss: on 3,014 fresh answers of the same
  families, +1.63 points [+0.70, +2.52].
• The git guard, larger set, served, paired with Ekbasis-27B (exact McNemar): false alarms on the held-out command
  types 26 of 799 against 49 (p = 5.6 × 10⁻⁶); work-losing commands flagged as often (776 against 767 of 799, p = 0.15;
  known types 837 against 828, p = 0.12); false alarms on the known types 27 against 37 of 863, within chance (p =
  0.076). "Held-out" means held out of the git sandbox's training rows: Skopos' factory rows contain git rebase <branch> and git restore --source=<ref> <file>, so for Skopos these command types are not unseen.
• Real-repository scenarios of Ekbasis-27B's release (T3c; not fresh): work-losing flagged 3 of 3, false alarms 1 of
  13, "will it fail?" right 17 of 17 (Ekbasis-27B 3 of 3, 1 of 13, 16 of 17).
• Confident errors: in world families it never saw, more of its wrong answers come with confidence ≥ 0.9, +0.75 per
  100 answers [+0.28, +1.23], and its ECE there is higher, +0.009 [+0.003, +0.015] (computed after the independent
  review; registered in the addendum but missing from the first report).
• In-distribution sets: executed factory scenarios 93.9% against 88.1% (3,878 items, without the 105 the overlap
  audit flags). R1 → R2 → R4 were selected on fac_val, git_long and git_long_val, drawn from the same generators
  as their training rows; the more independent evidence is the fresh never-trained items above (families never in
  training rows, though used on validation data to choose the first Skopos' interpolation) and the real agent turns of
  AgentWorldBench (AUROC of P(fail) 0.843 against 0.833 on 455 turns).
• Images (text-only training): preserved by the registered reading, +1.01 points [−2.03, +4.05] on 395 items with
  the state as an image (without the 5 items the overlap audit flags; preserved on all items too).
• Builds: FP8 (30.4 GB) and INT4 (18.6 GB) pass Ekbasis-27B's quantization gate. The MLX build misses 3 of its
  checks and is not part of this release until it passes its test on Apple hardware.
• Certified mode (ekbasis.verify, thresholds unchanged), pre-registered carry-over check: INT4 7 errors among 927
  accepted answers (95% upper bound 1.41%), bf16 6 among 935 (1.26%); Ekbasis-27B on the same scenarios 5 among 886.
• Overlap with the training data: every evaluation set was audited against every training row of the Ekbasis-27B
  stages, v49f and R4 (not Eikos-27B's own data or the Qwen base's pretraining), and every published number recomputed
  without the flagged items; no conclusion changes (results_skopos/overlap_audit/). Each table of the report on a set
  with flagged items says which numbers are on all items and which without them; for the confident-errors test, all
  items are the registered primary.

A pre-launch release blocker is declared (release decision): on
paraphrased short checks Skopos is right on 113 of 120 against Ekbasis-27B's 117 (−3.3 points, paired 95% CI [−6.7,
−0.8]), so the pre-registered rule of the pre-launch evaluation fired; the exact test on the 4 discordant items does not
reject (p = 0.125), and Ekbasis-27B re-run on the same machine reproduced its published answers (240/240).

Every result is reported, favourable or not; the deviations from the pre-registrations are listed at the end of
RELEASE_EVAL_SKOPOS.md.

How to check

python3 scripts/write_release_eval_skopos.py --final   # rebuilds RELEASE_EVAL_SKOPOS.md from results_skopos/; no diff
python3 results_skopos/scripts/verify_summaries.py <ekbasis-skopos-data raw/> --awb <AgentWorldBench suite rebuilt>
python3 training/skopos/stage_files_skopos.py <out dir>   # every stage's training file from ekbasis-skopos-data

generators/README.md gives a container recipe for running the generators, which execute the commands they generate.

🤖 Generated with Claude Code