Bayesian Optimization from Scratch

A Bayesian optimizer built with only NumPy and SciPy (Gaussian Process + Upper Confidence Bound) that tunes the regularization strength α of a Ridge Regression, animated step by step.

Motivation

This project started as a study exercise: understand how a Bayesian optimizer works when it is used to tune a machine learning model. Hyperparameters such as the regularization strength of a Ridge Regression are usually tuned with off-the-shelf libraries, which hide what happens inside. Here the optimizer is built by hand (a Gaussian Process surrogate plus a UCB acquisition function), and applied to a small, fully understood problem, tuning Ridge's α, so every step can be inspected, plotted and explained. The same ideas carry over to tuning larger models such as gradient boosting or neural networks.

How to read the animation. Top left: the dashed line is the true validation score (which the optimizer cannot see), the blue line is the GP's belief (μ) with its uncertainty band (μ ± β·σ), black dots are the α values evaluated so far, the red ring is the latest one and the gold star is the best so far. Bottom left: the UCB acquisition curve. Its peak (red triangle) is the α that will be evaluated next. Right: a table that gains one row per iteration. The red row is the α being chosen right now (its score is still "?"), and green rows are iterations that found a new best. The first iterations are held longer on screen because that is where the model changes the most.

---

The problem

Training a model to score one hyperparameter value can be slow or costly, so every evaluation should count. Instead of trying values blindly, Bayesian optimization builds a cheap model of the score from the evaluations so far and uses it to decide where to look next.

Here the "expensive function" is: fit Ridge with regularization α on synthetic data, return the validation MSE. The data (see src/data.py) is generated from scratch with controlled noise, so there are no external downloads.

How it works

Piece	Formula	Intuition
Ridge regression	w = (XᵀX + αI)⁻¹ Xᵀy	Small α overfits, large α underfits, so a sweet spot exists.
RBF kernel	k(x, x') = σ_f² · exp(−(x−x')² / 2ℓ²)	Nearby α values give similar scores. ℓ (bandwidth) says how near is "near".
GP posterior mean	μ(x) = K*ᵀ (K + σ_n²I)⁻¹ y	Our best guess of the score at any α.
GP posterior variance	σ²(x) = k(x,x) − K*ᵀ (K + σ_n²I)⁻¹ K*	How unsure we are. It is ~0 at evaluated points and grows away from them.
UCB acquisition	a(x) = μ(x) + β · σ(x)	Optimism under uncertainty: pick the α that could be best.

Exploration vs exploitation is controlled by β:

β	Behaviour
≈ 0	Pure exploitation: always samples the current best guess, can get stuck.
≈ 2	Balanced (default in this repo).
≥ 5	Exploration-heavy: spends evaluations on unexplored corners.

Two implementation choices worth knowing:

• We maximize −MSE. UCB is a maximization rule, and minimizing a loss is the same as maximizing its negative. The plots therefore show "validation score (−MSE), higher is better".
• We search over log₁₀(α) (from 10⁻⁴ to 10³). α spans orders of magnitude, so this makes a single kernel bandwidth meaningful across the whole range.

Quickstart

git clone https://github.com/lucasmonero/bayesian-optimization.git
cd bayesian-optimization
python -m venv .venv && source .venv/bin/activate    # Windows: .venv\Scripts\activate
pip install -r requirements.txt

python run_demo.py            # writes assets/bo_ucb_demo.gif and assets/bo_vs_random_grid.png
python run_demo.py --beta 5   # try a more exploratory optimizer
pytest                        # run the tests

The demo takes about 30 seconds, most of it rendering the GIF.

Project structure

src/
  data.py              synthetic data, closed-form Ridge, the black-box objective
  gaussian_process.py  GaussianProcessFromScratch: RBF kernel, prior, posterior
  bayes_opt.py         BayesianOptimizerUCB: the loop + UCB acquisition
  baselines.py         random & grid search for comparison
  visualization.py     per-iteration frames, GIF builder (Celluloid), comparison plot
notebooks/
  01_gaussian_process_intuition.ipynb        prior samples, posterior, bandwidth effect
  02_bayesian_optimization_walkthrough.ipynb end-to-end run, β experiments, extras
tests/                 GP checked against a naive matrix inverse; optimizer sanity checks
run_demo.py            one command to reproduce all assets

Results

Evaluations needed to get within a tolerance of the best possible validation MSE (same seed, β = 2, 3 random initial points):

Method	within 5%	within 1%
Bayesian optimization (UCB)	6	6
Random search	8	not reached in 15
Grid search	11	11

Over 20 different seeds with a budget of 15 evaluations, Bayesian optimization ended closer to the optimum than random search in 20 of 20 runs (mean MSE gap of about 0.0003 vs about 1.96 for random search).

Honest caveat: this is a one-dimensional problem, where even random and grid search do reasonably well, and absolute MSE differences are small. The advantage grows with dimensionality and evaluation cost.

Key takeaways: why Bayesian optimization beats Random or Grid search

• It learns from history. Grid and random search ignore every past result. Bayesian optimization uses all of them to choose the next point.
• It is uncertainty-aware. It explores only where it is still unsure and concentrates evaluations where the score looks promising.
• It scales better. A grid needs exponentially more points as you add hyperparameters, while a surrogate model spends its budget adaptively.
• Trade-offs. The GP costs O(n³) in the number of evaluations, and results depend on the kernel and β. It shines when evaluations are expensive and dimensions are few (roughly under 20).

Experiments to try

• Change --beta (0.1, 2, 8) and watch how the search behaviour changes.
• Change the kernel length_scale in GaussianProcessFromScratch (see notebook 01).
• Enable optimize_hyperparams=True to learn ℓ and σ_f from the data (notebook 02).
• Change the difficulty of the data in generate_regression_data (noise, number of features).

References

• Rasmussen & Williams, Gaussian Processes for Machine Learning, ch. 2.
• Srinivas et al., 2010, Gaussian Process Optimization in the Bandit Setting (GP-UCB).
• Brochu, Cora & de Freitas, 2010, A Tutorial on Bayesian Optimization.

License

MIT