An engineering investigation into budgeted high-fidelity evaluation for symbolic regression: if fitting every candidate equation well is too expensive, can a cheaper curvature-aware score be substituted without degrading the search?
The answer we measured is no, for a specific and reproducible reason. This repository documents that result, the instrumentation built to obtain it, and the components that remain useful independently of it.
| Question | Verdict |
|---|---|
| Does a one-step Gauss-Newton score beat raw loss at equal evaluation budget? | Yes (+0.217 recovery, CI [0.05, 0.40]) |
| Does it beat native PySR at equal wall-clock? | No (−0.136, CI [−0.304, 0.000]) |
| Does more constant optimization help? | Yes — native > one-step > raw at every budget |
| Is the fused population evaluator numerically correct? | Yes (~1e-13 vs analytical reference) |
| Does batched constant fitting match SciPy quality? | No — 7–11× faster, 5–11% worse fit |
The investigation is closed. Nothing here is presented as state of the art.
Symbolic regression is bi-level: an outer loop proposes discrete equation structures, an inner loop fits their continuous constants. The inner loop's result becomes the structure's fitness, so a structure that is correct but badly fitted can be discarded — the "good structure, bad score" failure.
The obvious fix is to fit every candidate better, which is expensive. This project tested the complementary idea: spend the fitting budget only where it can change a selection decision, using a cheap curvature-aware proxy elsewhere.
Two independent lines, both instrumented against a real host (PySR 1.5.10 / SymbolicRegression.jl).
Offline selection study. A CPU reference with analytical Jacobians, a multistart high-fidelity oracle, split-conformal calibration at the level of whole selection events, and a learned post-refinement surrogate. Frozen protocols with untouched confirmation seeds.
In-search host integration. A non-mutating Gauss-Newton objective in Julia
(refinegate/julia/), qualified to 2.6e-17 agreement with the CPU reference,
plus a passive population-snapshot logger qualified to leave search results
byte-identical.
The Gauss-Newton objective was first confirmed at an equal host-evaluation budget — 180 runs, 6 families, 10 untouched seeds:
| arm | recovery | median oracle MSE |
|---|---|---|
| Gauss-Newton lookahead | 0.333 | 0.00123 |
| native PySR | 0.117 | 0.0168 |
| raw (no constant fitting) | 0.100 | 0.0324 |
Gain over native: +0.217, task-clustered 95% CI [0.05, 0.40].
That framing turned out to be misleading. max_evals does not govern compute:
with Julia startup amortized, PySR evaluations cost ~0.010 ms each, so the
baseline can simply buy far more of them. Re-running on matched problems under a
wall-clock budget (22 problems, 6 tasks, common support):
| arm | recovery | median wall-clock |
|---|---|---|
| Gauss-Newton @ 32 768 evals | 0.591 | 25.0 s |
| raw @ 262 144 | 0.455 | 13.8 s |
| native @ 262 144 | 0.636 | 14.6 s |
| raw @ 1 048 576 | 0.591 | 23.3 s |
| native @ 1 048 576 | 0.727 | 22.4 s |
Paired differences (treatment − baseline): −0.045 vs native@262k;
−0.136, CI [−0.304, 0.000] vs native@1M; +0.000 vs raw@1M.
Conclusion. Native PySR matches the treatment's recovery in 58% of the wall-clock and beats it at equal time. When candidate evaluation is this cheap, evaluating more candidates dominates evaluating them more cleverly.
The arm ordering — native > one-step > raw at every budget — also says the premise was backwards: more constant optimization helps, and substituting a cheap proxy for real fitting costs accuracy.
Recorded because they were load-bearing assumptions at the time.
| Hypothesis | Outcome |
|---|---|
| Gauss-Newton scoring is ~220× costlier per evaluation | False. Startup-contaminated fit. True marginal cost ~1.3×; the real penalty is a one-time ~12.6 s objective compilation. |
| Fused evaluation advantage grows with dataset size | False, inverted. 47.3× at N=128 → 0.4× (slower) at N=32 768. Fusion amortizes per-expression Python overhead only. |
| Conformal intervals can certify selection decisions | False. Event radius ~18.9 log units; 264/280 candidates still required refinement. |
| Nielsen gain-ratio damping improves batched LM | False. −0.140 vs −0.160 for the simpler fixed schedule. Reverted. |
Fused population evaluator (refinegate/fused_population.py) — a padded
postfix batch interpreter with forward-mode derivatives computing per-program
loss, Jᵀr, and JᵀJ without materializing a [programs, samples, parameters]
Jacobian. Validated on 312 of 313 real captured PySR expressions, matching the
per-expression analytical reference to ~1e-13.
| population | per-expression loop | fused | speedup |
|---|---|---|---|
| 8 | 3.100 ms/cand | 0.3911 | 7.9× |
| 128 | 2.952 | 0.0737 | 40.0× |
| 256 | 3.108 | 0.0607 | 51.2× |
| 2048 | 4.923 | 0.1050 | 46.9× |
Best at P=128–512; degrades beyond as the derivative stack exceeds cache.
Batched constant fitting (population_levenberg_marquardt) — whole-population
Levenberg-Marquardt with per-candidate step acceptance, damping, and early
termination. Against per-expression SciPy least_squares with analytical
Jacobians:
| config | speedup | SciPy mean log₁₀ loss | batched |
|---|---|---|---|
| P=128, 64 steps | 10.7× | −0.206 | −0.160 |
| P=256, 64 steps | 7.2× | −0.161 | −0.141 |
This is a trade, not a win: ~7–11× throughput at ~5–11% worse fit quality.
Passive snapshot logger (refinegate/pysr_snapshots.py) — captures real PySR
population distributions with stable member identities via
AbstractSRLogger.logging_callback!, without patching search. Qualified across
two seeds in fresh processes: hall-of-fame and final populations byte-identical
with the sink on and off.
Offline selection result — a structural-abstention policy (use the one-step
score when the best cheap candidate is conditionally linear, else a learned
post-refinement surrogate) improved top-k recall by +0.0505, CI
[0.0253, 0.0760], over 80 untouched tasks and 1 920 tournaments, with zero
online high-fidelity fits and 82.5% simultaneous population coverage.
The original protocol required every generated candidate to belong to exactly one
selection pool. PySR's native tournaments violate this by construction: one
iteration produces ≥90 tournament memberships across ≤44 distinct candidate
identities, so reuse is unavoidable (pigeonhole). v1 therefore terminated at
STOP_PROTOCOL_HOST_MISMATCH before any result-bearing run.
Frozen evidence: REFINEGATE_FINAL_DECISION.md,
QUALIFICATION_REPORT.md,
results/refinegate/v1/.
The v2 design removes that assumption: evidence is cached per candidate identity and reused across overlapping tournaments.
python -m pip install -e ".[dev]"
python -m pytest tests -q # 68 tests
ruff check refinegate experiments testsCPU-only components (fused evaluator, batched LM, offline study) run anywhere. Host-integration experiments additionally require PySR 1.5.10 and a working Julia toolchain; the first import provisions Julia packages and is slow.
# Fused evaluator vs per-expression scoring, on captured PySR expressions
python experiments/benchmark_fused_population_v2.py --output results/bench-fused
# Batched constant fitting vs SciPy least_squares
python experiments/benchmark_batched_constant_fitting_v2.py --output results/bench-lm
# Host study (requires PySR + Julia; hours)
python experiments/run_pysr_objective_pilot_v2.py \
--config configs/refinegate_v2/pysr_objective_confirmatory.json \
--output results/refinegate/v2/rerunAll host runners execute each job in a fresh process, reuse completed job artifacts on resume, and record failures rather than aborting a batch.
refinegate/ library
fused_population.py batched postfix interpreter, fused statistics, batched LM
expression_refinement.py strict AST parameterizer + analytical Jacobians
pysr_snapshots.py passive population logger and parser
julia/ non-mutating Gauss-Newton objective
conformal.py event-level split-conformal calibration
surrogate.py post-refinement surrogate + abstention policy
budget.py policies.py evaluation.py trajectory.py protocol v1 core
configs/refinegate_v1/ frozen v1 protocol
configs/refinegate_v2/ frozen v2 protocols (seeds fixed before execution)
experiments/ runners, qualifications, benchmarks
results/refinegate/v1/ frozen v1 stop evidence (checksummed)
tests/ 68 tests
- Single host. Only PySR 1.5.10 / SymbolicRegression.jl was instrumented.
- Narrow tasks. Six 1-D families. Rational and logarithmic recover at 0.00 for every arm — no method tested here solves them.
- Small data. N=128 samples. Conclusions about evaluation cost do not transfer to large-N regimes.
- CPU only. No GPU implementation; the fused evaluator is NumPy.
- Not integrated. The 51× scoring and 10.7× fitting speedups are measured on captured expressions, not demonstrated as an end-to-end search win.
- Bulk artifacts untracked. Only small frozen decision records are committed.
Findings are backed by machine-readable artifacts under results/refinegate/:
frozen configs with SHA-256 hashes, per-run records, paired analyses with
task-clustered bootstrap intervals, and gate decisions emitted after execution.
Protocols fixed their seeds and thresholds before the runs they judge. Gates that
failed are retained as failures.
Prior exploratory work (CurvSift / ArrayPull) was removed from the working tree during release cleanup; it remains in git history and its claims are superseded by the measurements above.
MIT — see LICENSE.