0
$\begingroup$

In this cs231 lection note there is a counterintuitive quote:

The gap between the training and validation accuracy indicates the amount of overfitting.

...

The other possible case is when the validation accuracy tracks the training accuracy fairly well. This case indicates that your model capacity is not high enough: make the model larger by increasing the number of parameters.

enter image description here

Questions:

  1. Isn't the goal of training to achieve the validation accuracy as high as training accuracy or did I miss something?
  2. What the intuition under increasing the number of parameters if the validation accuracy tracks the training accuracy fairly well?
$\endgroup$
1
$\begingroup$

I think this:

The other possible case is when the validation accuracy tracks the training accuracy fairly well. This case indicates that your model capacity is not high enough: make the model larger by increasing the number of parameters.

Is getting at the idea that training a model, in general, should yield overfitting. That is basically its design -- tweak weights until error is zero.

So, if your model is NOT overtly overfitting, then your model can probably "learn more" with more trainable parameters which in turn, should, help you get even better validation metrics.

$\endgroup$
2
  • $\begingroup$ I got the idea. Thanks Are there any techniques to solve the trade-off between min training loss and min train/val difference in loss? IOW, when should I stop tweaking the weights? $\endgroup$ – discort Feb 4 at 14:15
  • $\begingroup$ You should freeze weights when you maximize or minimize whatever training objective you have. If it is loss, then whenever you reach the minimum. If it is F1, then whenever you reach the maximum. $\endgroup$ – John Stud Feb 4 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.