I was solving this question about tuning hyperparameters and I don't understand how to choose the number of hyperparameters by using the training error (TE) and the validation error (VE). Define the variance as VAR=TE$-$VE.
The question is the following:
You are tuning a hyperparameter for an algorithm. The following table shows a data set with different hyperparameter, training error, and validation errors.
|Hyperparameter (H)||Training error (TE)||Validation error (VE)|
- Which value of H should you select based on the data?
- What H value displays the poorest training result?
The suggested answer is H=4 and H=5 respectively because H=4 minimizes the variance (variance is 5) and H=5 maximizes the variance (variance is 350).
However, I do not see how minimizing the variance is responsible for tuning the "correct" number of hyperparameters. For me it would make more sense to select the value of H which minimizes the VE for 1. (H=1) and the value of H which maximizes the VE for 2. (H=3 or H=4?).
Can someone please clarify what should be the correct choice and why?