I was solving this question about tuning hyperparameters and I don't understand how to choose the number of hyperparameters by using the training error (TE) and the validation error (VE). Define the variance as VAR=TE$-$VE.

The question is the following:

You are tuning a hyperparameter for an algorithm. The following table shows a data set with different hyperparameter, training error, and validation errors.

Hyperparameter (H) Training error (TE) Validation error (VE)
1 105 95
2 200 85
3 250 100
4 105 100
5 400 50
  1. Which value of H should you select based on the data?
  2. What H value displays the poorest training result?

The suggested answer is H=4 and H=5 respectively because H=4 minimizes the variance (variance is 5) and H=5 maximizes the variance (variance is 350).

However, I do not see how minimizing the variance is responsible for tuning the "correct" number of hyperparameters. For me it would make more sense to select the value of H which minimizes the VE for 1. (H=1) and the value of H which maximizes the VE for 2. (H=3 or H=4?).

Can someone please clarify what should be the correct choice and why?


1 Answer 1



Your reasoning is correct. Whenever we pick a model (to include selecting hyper-parameters), we are balancing bias and variance. The optimal hyper parameter will minimize the error created by both of these effects.

Here's an article that digs a little deeper into the subject - https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html.

Take a look at the bias-variance tradeoff (Validation Curve Schematic) graphic about halfway down.

You might want to read a little about cross-validation which is another way to identify optimal hyper-parameters.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.