README: cross-family review up front, the overseer, numbers from the real ledger - #10
Conversation
…ch с заложенными багами README переписан: первые строки продают ревью другой семьёй на своих подписках, голос персонажа (прораб с двумя бригадами), Before/after на одном примере, раздел Numbers с честными результатами эксперимента и счётчиками тестов, схема потока (SVG, светлая и тёмная), Install по хостам, таблица команд, FAQ с «кому не подходит». Ссылка T3 Code ведёт на pingdotgg/t3code. bench/seeded-review: четыре диффа с двумя заложенными дефектами каждый, бриф, раннер, список дефектов и сырые результаты 8 прогонов от 2026-09-02 с ручной оценкой. CI прогоняет shellcheck по run.sh. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…откие строки Подпись лежала на пересечении стрелок, строка про SEQUENTIAL вылезала за рамку, стрелка от координатора упиралась в границу блока. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 054fc351ba
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -0,0 +1,97 @@ | |||
| diff --git a/review/01-pagination.diff b/review/01-pagination.diff | |||
There was a problem hiding this comment.
Starting with case 02, applying this file also adds the entire previous benchmark diff as review/01-pagination.diff; cases 03 and 04 recursively include all earlier cases in the same way. Reviewers are therefore given progressively larger changes containing defects from other test cases, despite the benchmark describing each input as one isolated source/spec/test change. A reviewer can report or be influenced by earlier planted defects, corrupting per-case scoring and timing, so regenerate cases 02–04 directly against the clean base.
Useful? React with 👍 / 👎.
| wt="$WORK/wt/$n"; git worktree add -q "$wt" "$n" | ||
| sed -e "s#__NAME__#$name#g" -e "s#__WT__#$wt#g" -e "s#__DIFF__#$diff#g" "$HERE/brief.md" > "$WORK/$n.brief.md" | ||
| for backend in claude codex; do | ||
| id="$(node "$AR" run --role reviewer --backend "$backend" --cwd "$wt" --brief "$WORK/$n.brief.md" --detach --timeout 25 </dev/null | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>console.log(JSON.parse(s).id))')" |
There was a problem hiding this comment.
When any of these eight concurrent reviews hits a usage or rate limit, this invocation uses agent-run's default automatic fallback. The checked wrapper at skills/delegate-kit/scripts/agent-run:1031-1043 marks the requested run failed-quota and launches the other backend under a new ID, while this script waits for and copies only the original ID. The fallback result is therefore lost and the named family has no usable result, undermining the family comparison; pass --fallback none for these benchmark runs.
Useful? React with 👍 / 👎.
| cp "$HOME/.delegate-kit/runs/$3/result.json" "$OUT/$1-$2.json" 2>/dev/null || echo "no result for $1 $2 ($3)" | ||
| echo "$1 $2: $(node -e 'const r=require(process.argv[1]);console.log(r.findings.length+" findings")' "$OUT/$1-$2.json" 2>/dev/null || echo failed)" |
There was a problem hiding this comment.
When the output directory already exists—normally when rerunning the benchmark on the same day—and a worker produces no result.json, the failed cp leaves the previous $OUT/$1-$2.json untouched. The next line then parses that stale file and reports its finding count as if it came from the current run, silently mixing experiments. Delete each destination before launch or fail the run when collection fails.
Useful? React with 👍 / 👎.
tomastaker
changed the title
…п в стиле ponytail Персонаж заменён на египетского надсмотрщика с плетью и папирусом; схема и текст переведены на «gangs» и «tally». Раздел Numbers больше не цитирует пилот с заложенными багами (обе семьи нашли 8 из 8, преимущества он не показывает): вместо него таблица из ledger автора за 20.08–01.09.2026, 46 кросс-вендорных ревью реальных диффов, 17 с high-находкой. bench/ledger-stats.sh печатает ту же таблицу из любого ledger. Логотип: чёрно-белый рисунок, вырезан на прозрачный фон, светлый и тёмный. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
README: cross-family review up front, the overseer, numbers from the real ledger
1 participant
Problem
The README led with "shape first", buried the one thing no other skill offers (the other model family reviews the diff, on subscriptions you already have), linked T3 Code to the chat product, and carried a "shared blind spots" claim with no evidence behind it.
Solution
Numbers,How it workswith a light/dark SVG flow, per-host install, a commands table, FAQ including "who this is not for". T3 Code badge now points at pingdotgg/t3code.Numberscome from the author's real ledger (20 Aug – 1 Sep 2026, ten repositories): 46 cross-family reviews of real diffs, 17 with at least one high-severity finding, medians per family.bench/ledger-stats.shprints the same table from any ledger, so the numbers are reproducible per user.assets/: hand-cut logo (light and dark), two SVG diagrams.bench/seeded-review/: a small planted-defect harness (4 diffs × 2 reviewers) with its pilot results. Both families caught 8/8 planted defects, so the pilot is kept as a harness, not cited as evidence in the README.Checks
delivery.sh --race 20: 0 duplicates, 0 lost.bench/ledger-stats.shreproduces every number in the README table from the local ledger.run.shdry-run against a stubagent-run: repo build, branches, worktrees, launches and collection execute.Limits
🤖 Generated with Claude Code