frontier-cs.org

The source of frontier-cs.org, the website of the FrontierCS benchmarks. It is a Jekyll site, originally forked from the al-folio theme and since rebuilt.

• FrontierCS 2 evaluates AI agents on computer science research. Each of its 48 tasks removes the key contribution of a published paper from the code released with it. An agent passes only if its version beats that code, the human reference, on every metric of every hidden workload.
• FrontierCS is the first benchmark: open-ended algorithmic problems where any valid answer is accepted and a better one scores higher.

Pages

Path	Source	Contents
/	index.md, assets/js/fcs2.js, assets/css/fcs2.css	FrontierCS 2: the task map, the leaderboard (pass rate, pass rate by domain, FCI, test-time scaling, mean score), the task list, the private set, sponsors and questions
/task/#<slug>	_pages/fcs2-task.md, assets/js/fcs2-task.js	One FrontierCS 2 task: its diagram and description, results, pass criteria, charts and the solution in words. /task/ alone goes to the task list
/v1/	_pages/v1.md, assets/js/fcs1.js	FrontierCS: leaderboard, example problems and questions
/blog/	_pages/blog.md, _posts/	Blog list and posts
/paper/	_pages/projects.md	Research: the team's papers and work built on FrontierCS
/workshop/	_pages/workshop.md	The proposed ICLR 2027 workshop
/team/	_pages/team.md	Publications, institutions, partners, sponsors and contacts
/results2/	_pages/results2.html	An unlisted results page; see bin/README-results2.md

Shared parts are in _includes/ (header, footer, the blog article frame) and _layouts/. Site-wide styles and colour tokens are in _sass/_fcs.scss.

Run it locally

bundle install
bundle exec jekyll serve          # http://localhost:4000

or, with Docker, ./bin/docker_run.sh (http://localhost:8080). A production build is JEKYLL_ENV=production bundle exec jekyll build, which writes _site/.

Branches and deployment

• main is the live site. A push to main runs .github/workflows/deploy.yml, which builds the site and publishes it to the gh-pages branch; CNAME serves it at frontier-cs.org.
• Start each change on a new branch from main and open a pull request against main, one topic per pull request, with a preview and what you checked (desktop and a 390 px phone width).

FrontierCS 2 data

The FrontierCS 2 pages read static files; nothing on the site calls the evaluation platform. The files come from the platform's read-only API through these scripts, run in order:

1. bin/fcs2_db_from_platform.py out.db --claude-usage=opus55-usage-recovered.json collects the formal evaluations into a SQLite database. The usage file restores Claude Opus 5.5's costs, which the platform lost when traces were truncated. Without it the script stops while any Claude Code run has no usage. Also pass --trace-usage=fcs2-trace-usage-recovered-20261022.json (in exports/trace-usage/), the costs of Progent's runs and Gemini 4 Argon's Continuum run; bin/fcs2_db_add_trace_usage.py adds it to an existing database. The first 13 models' evaluations are on no platform since 2026-10-24. bin/fcs2_db_from_release.py out.db formal-results.json 99df6b5 rebuilds their snapshot from the public release's run table (metadata/formal-results.json of the Hugging Face dataset FrontierCS2/FrontierCS-2-runs) and the site's own published files at that commit; the converter reproduces that commit's data from it. When a newer release changes records (reruns, re-judged finals), run it with REV = the current main and --spend=bin/fcs2_release_spend.jsonl, the spend and development scores of the changed runs, which main's TRAJ does not hold; step 2 is then bin/fcs2_checks_from_release.py formal-results.json BASE.json, which sets the changed records' PASSBY from the release's checks. A model run on the new platform is collected alone with --only='<candidate name>' (and --run-cost= for its provider-billed costs) (GPT-6.1 Terra and Kimi K3.1 Pro, billed by OpenAI and Kimi directly, have no gateway cost: their --run-cost files price the token usage of every agent request in the run's evidence at the provider's list price; Claude Fable 5.1, run on a Claude subscription, takes from its evidence the cost Claude Code reports for itself; Gemini 4 Carbon's cost is estimated at Gemini 4 Argon's prices) and added with bin/fcs2_db_add_snapshot.py out.db new.db; step 2 then takes the same --only=. A candidate whose 48 runs span several evaluations is listed in the script's EVALUATIONS; a run that only judged another run's stored final (a FINAL-only recovery) is in ORIGIN, which gives the run the agent worked in for time, spend and development scores. WHY records, per run in ZERO, why it ended without a final; the kebab shows it.
2. bin/fcs2_checks_from_platform.py out.db writes assets/js/fcs2-checks.js: for each run and metric, on how many hidden workloads the final submission beat the human reference. Task pages and the overview's Every run kebab read it. The counts of models kept off the leaderboard (Kimi K2.7) go to _data/fcs2_checks_hidden.json instead, for fits only; the converter reads both.
3. bin/fcs2_runs_from_db.py out.db rewrites the block between the converter markers in assets/js/fcs2-data.js: runs, costs, FCI, test-time scaling, task difficulty, how each run ended when it failed, was refused or is not yet scored (STATE, for the kebab) and why, where step 1 recorded it (STATE_WHY, the kebab's tooltip), and descriptions. Task difficulty reads assets/js/fcs2-checks.js and _data/fcs2_checks_hidden.json, so step 2 comes first. It also writes _data/fcs2_runs_hidden.json, the runs of models kept off the leaderboard (Kimi K2.7, the 0 point of FCI). Files in _data/ are not published. With --json=out.json it also writes every number for the paper's figures, and keeps there the earlier FCI fitted on raw scores (feci_score), which the site no longer shows.
4. bin/fcs2_metric_means.py metric_aggregates.jsonl --db=out.db --values=metric_values.jsonl --margins=workload_margins.jsonl --reference=reference_profile.jsonl --reference=reference_profile_polar.jsonl --reference=reference_profile_replay.jsonl writes assets/js/fcs2-metrics.js: for each task and metric, whether higher or lower is better and, per run, its mean over the hidden final workloads and the human reference's mean in the same evaluation. The input is an export of each run's final evidence from the platform (GET /api/v1/runs/<id>/submission/evidence?submission_id=final, one row per run and metric). The human reference is measured again in every run, so its mean varies from run to run; the task page shows the median over the task's runs. Where no run's evidence holds the human reference's numbers, --reference gives them: one row per task and metric, measured in a run of the human reference alone or read from earlier evidence, used only on the task version the runs had. The export comes from bin/fcs2_metric_values_from_evidence.py RELEASE_DIR OUT_DIR, which reads the final verifier evidence in the public release (FrontierCS2/FrontierCS-2-runs, traces/<task>/<model>/evidence.tar.zst) with one reader per task in bin/fcs2_metric_parsers/<task>.py; then run step 4 with --reference=bin/fcs2_metric_reference_profile.jsonl (the profiled references as the page showed them on 2026-10-22). bin/fcs2_metric_carry.jsonl keeps the few numbers the released evidence does not hold (SCOPE's batch latency, BBQ's throughput and latency).
5. bin/fcs2_patches_from_platform.py writes assets/patches/<slug>.json, the final patch of each run. The site does not show patches; they are the source for the solutions below.

These files are written by hand:

• assets/js/fcs2-data.js, outside the converter block: the task list, domains and shared helpers.
• bin/fcs2_descriptions.tsv: task descriptions, a copy of the paper repository's notes. The converter copies them into fcs2-data.js.
• assets/js/fcs2-solutions.js: how a model's final submission solves each task, in 3 to 5 steps, written from its patch.
• assets/img/fcs2-tasks/<slug>.svg: one diagram per task, inlined into the page. Diagrams use only the fig2 classes in assets/css/fcs2.css, never inline colours or fonts. What is given sits on grey, the part the agent writes on the tonal colour of the task's domain.
• assets/js/fcs2-art.js: the illustration of each task in the map.

Definitions

• Pass: the final submission beats the human reference on every metric the task checks and on every hidden workload.
• FCI (FrontierCS Capabilities Index) and FDI (FrontierCS Difficulty Index) come from one model, Epoch AI's ECI idea fitted on what each run achieves. A run passes only if it beats the human reference on every metric the task checks, so the model looks at each metric (from assets/js/fcs2-checks.js); metrics that the same runs beat share one difficulty (GROUP_TOL in the converter). A run's final score counts by how far above the human reference it is when the run passes, in units of the task's score spread (at least the median spread over tasks), and by how far below it when the run does not pass and falls short; a run that does not pass but scores above the human reference counts as at least level, as its metrics did not confirm more. A run's metrics and its final score come from the same measurements, so they share one fitted run effect. Every run of every model, Kimi K2.7 included, enters the fit.
  • A model's FCI is its strength, with Kimi K2.7 at 5 and ten points per logit (PASS_ZERO, PASS_ZERO_VALUE and PASS_UNIT in the converter). Its 90% interval comes from a bootstrap over tasks that refits the model.
  • A task's FDI is the FCI at which a model passes it half the time, with a 90% interval. The task chart draws each task's pass curve from its metric groups' difficulties (GROUPS). With --json the converter also writes a progress curve per task, the expected share of its hidden checks a model beats (progress_curve).
• Refusals: a run whose model provider refused, so that no final submission came (the platform's adjudication cause provider_refusal_no_final), counts as a failed run everywhere, in FCI, FDI and test-time scaling as in the pass rate (Qiuyang, 2026-10-20). The task page lists it as refused.
• Test-time scaling: a model's FCI with every run stopped at a budget in US$. A run that has ended within the budget counts what it achieved; a run still going counts as if it timed out at the budget (every metric group unbeaten, a final score of 0), so a curve never falls. The chart shows only each lab's model with the highest FCI (owner, 2026-10-16). Task difficulties stay as in the leaderboard fit, so at full budget it is the model's FCI.
• Cost: how each model's cost is estimated is in the paper (Appendix B, Cost estimates); the site's FAQ no longer carries it (owner, 2026-10-20).

Images for sharing

These need Python with Playwright; each script's docstring gives its use.

• bin/fcs2_card.py draws assets/img/fcs2-card.png, the link preview of the FrontierCS 2 page.
• bin/fcs2_thread_cards.py draws post cards from the site's charts.
• bin/blog_thumbs.py draws the blog thumbnails in assets/img/blog-thumbs/.
• bin/blog_figures.py redraws figures used in blog posts.

Write a blog post

1. Add _posts/YYYY-MM-DD-<slug>.md with layout: distill. Copy the front matter of a recent post: title, description, date, date_display, authors (name, url, affiliations), thumbnail, og_image, and toc for the contents list.
2. Put images in assets/img/YYYY-MM-DD-<slug>/.
3. Add a thumbnail with bin/blog_thumbs.py, or point thumbnail at an existing image.

Conventions

• Write "FrontierCS" (no hyphen) and "FrontierCS 2"; the hyphen stays only in repository names and URLs. The authors' code is the "human reference".
• Colours come from Google's palettes. Each model's colour is defined once, as --m-<id> in _sass/_fcs.scss.
• Links are plain underlined text or pills, never arrow characters. Checkboxes, radios and menus are drawn in the site's style, not left to the browser.
• Check every change at desktop width and at 390 px.

License

MIT, see LICENSE. The theme this site started from is al-folio by Maruan Al-Shedivat and contributors, also MIT.