How to choose a model's hyperparameters in terms of the variance?

Question

I was solving this question about tuning hyperparameters and I don't understand how to choose the number of hyperparameters by using the training error (TE) and the validation error (VE). Define the variance as VAR=TE$-$VE.

The question is the following:

You are tuning a hyperparameter for an algorithm. The following table shows a data set with different hyperparameter, training error, and validation errors.

Hyperparameter (H)	Training error (TE)	Validation error (VE)
1	105	95
2	200	85
3	250	100
4	105	100
5	400	50

1. Which value of H should you select based on the data?
2. What H value displays the poorest training result?

The suggested answer is H=4 and H=5 respectively because H=4 minimizes the variance (variance is 5) and H=5 maximizes the variance (variance is 350).

However, I do not see how minimizing the variance is responsible for tuning the "correct" number of hyperparameters. For me it would make more sense to select the value of H which minimizes the VE for 1. (H=1) and the value of H which maximizes the VE for 2. (H=3 or H=4?).

Can someone please clarify what should be the correct choice and why?

Answer 1

Babado,

Your reasoning is correct. Whenever we pick a model (to include selecting hyper-parameters), we are balancing bias and variance. The optimal hyper parameter will minimize the error created by both of these effects.

Here's an article that digs a little deeper into the subject - https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html.

Take a look at the bias-variance tradeoff (Validation Curve Schematic) graphic about halfway down.

You might want to read a little about cross-validation which is another way to identify optimal hyper-parameters.