0
$\begingroup$

You have $N$ observations of a function $F$ and you want to get an approximation $G$ of the function $F$. The type of model $G$ that you selected depends on a hyperparameter $l$. To determine the hyperparameter $l$, you split your data into three sets:

  • The test set, made up of $N_T$ observations and denoted by $(\mathbf{X}_{N_T}, \mathbf{Y}_{N_T})$, is the unseen data so you do not touch it;
  • The validation set, made up of $N_V$ observations and denoted by $(\mathbf{X}_{N_V},\mathbf{Y}_{N_V}$);
  • The training set, made up of $N_L$ observations and denoted by $(\mathbf{X}_{N_L},\mathbf{Y}_{N_L}$).

Let's focus on the validation set and the training set. On the training set, we use marginal likelihood estimation to determine the hyperparameter $l$. That is, you minimize $p(\mathbf{Y}_{N_L}| \,\mathbf{X}_{N_L}, \,l)$ with respect to the hyperparameter $l$ through a minimization algorithm. You can plot the value of $p(\mathbf{Y}_{N_L}| \,\mathbf{X}_{N_L}, \,l)$ with respect to the iteration of the minimization algorithm. Simultaneously, you plot the MSE on the validation test using the values of the hyperparameter optimized in the learning set. You can plot the MSE on the validation set with respect to the iteration of the minimization algorithm. In that way, you can notice if the model is overfitting the learning set or not. We can then select the hyperparameter $l$ which corresponds to the iteration where the MSE is minimal on the validation set, such as in early stopping method. Finally, we combine the training + validation sets and train the model $G$ on these data with the hyperparameter selected previously and compute the MSE on the test set to obtain a "true" approximation of the generalization error. Is it something one can do?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.