I was solving this question about tuning hyperparameters and I don't understand how to choose the number of hyperparameters by using the training error (TE) and the validation error (VE). Define the variance as VAR=TE$-$VE.
The question is the following:
You are tuning a hyperparameter for an algorithm. The following table shows a data set with different hyperparameter, training error, and validation errors.
Hyperparameter (H) | Training error (TE) | Validation error (VE) |
---|---|---|
1 | 105 | 95 |
2 | 200 | 85 |
3 | 250 | 100 |
4 | 105 | 100 |
5 | 400 | 50 |
- Which value of H should you select based on the data?
- What H value displays the poorest training result?
The suggested answer is H=4 and H=5 respectively because H=4 minimizes the variance (variance is 5) and H=5 maximizes the variance (variance is 350).
However, I do not see how minimizing the variance is responsible for tuning the "correct" number of hyperparameters. For me it would make more sense to select the value of H which minimizes the VE for 1. (H=1) and the value of H which maximizes the VE for 2. (H=3 or H=4?).
Can someone please clarify what should be the correct choice and why?