Hyperparameter Tuning

A hyperparameter sets a rule for making predictions. For a nearest-neighbor regressor, k is the number of training targets to average. Choosing k by training error alone can reward a model that merely reproduces its own examples. A separate validation set lets us compare candidates using predictions at other inputs. This module makes that comparison concrete: four candidates, seven fixed data points and a search budget you can replay.

Spend, inspect and rewind a real search

Open the Hyperparameter Tuning interactive module. On narrow screens the controls scroll above the results; the trial and neighbor tables have their own horizontal scroll. Compare the cards with the numerical tables, rather than estimating a score from bar height.

1. Press Reset search. Grid order is 1→2→3→4, Evaluated trials is 0, and Best k so far is None. The fixed training and validation data are still available. No untested candidate is presented as a result.
2. Press Next trial. Confirm best k = 1, validation MSE = 16/3 and training MSE = 0. At the default Validation query x = 3, inspect the ranked indices 1, 2, 0, 3 and prediction 0. Change the query to x = 5: the first index is 2 and prediction is 8.
3. Press Next trial again. Best k becomes 2, validation MSE becomes 0, and training MSE becomes 4. The three purple prediction bars now match the gray validation targets. Select Inspect candidate → k = 1: its earlier errors return in the inspector, while the best-result card stays k = 2.
4. Press Finish budget, then inspect k = 4. Its validation predictions are 4, 4, 4 and MSE is 32/3. The incumbent remains k = 2. Set Trial budget to 1: only the first trial remains eligible and the removed inspection falls back to Best so far. Increase the budget to 4; the evaluated count stays 1 until you request another trial.
5. Select Search order → Seeded shuffle and set Shuffle seed to 21. Each edit starts at zero evaluated trials. Press Next trial twice: the order is 3→1→4→2, and k = 1 wins the exact 16/3 tie. Set Evaluated trials to 3 and then 4 to see that k = 2 becomes eligible only at the final trial.
6. Keep shuffled order, enter seed 42, set budget to 2 and press Finish budget. Best k is 1. Raise the budget to 3: the two past scores remain unchanged. Press Next trial to admit k = 2 and obtain zero validation MSE. Press Previous trial to remove that candidate and restore the earlier incumbent.
7. Set Evaluated trials to 0 before trying playback. Set Playback speed to 2, press Play, then Pause before the budget is complete. Play advances one trial per half-second and stops at the budget. Manual query, inspection and speed changes pause without changing evaluated scores; changing order or seed starts a new search. Resize the page and confirm the chosen experiment remains intact.
8. Enter 1e1 in Shuffle seed. The text stays visible with an error and old search results disappear. Replace it with 001, a valid spelling of seed 1. The new order is 4→2→1→3 with zero trials evaluated. Reset search returns every control to its default.