3
$\begingroup$

It is common in Deep Learning to use SGD (with mini batches). However it is also common that the data sets are too large to compute the full current accuracy of the model using the whole data set as it trains. Thus, it seems that one alternative is to instead use fractions of the data set to asses how the training is going as SGD trains the model. It seems the reasons one would do this is because checking for every update on the full data set how the model is doing is ridiculously expensive in time $O(NT)$ where $N$ is the size of the data set and $T$ is the update iterations. It also seems unnecessary because as the $N$ increases it seems we only get sub linear gains on the accuracy. I base that because the standard dev of sample mean grows as $\frac{\sigma}{ \sqrt n}$ and not $\frac{\sigma}{ n }$, so for every doubling of the data set size we only gain an increased accuracy of $\sqrt 2$. So it takes twice as long to compute but we don't get an estimate twice as accurate but less.

Thus my question is about the best practices to asses how the model is doing using fractions of the train, validation and test set (specially in the context of deep learning).

What fraction of the Training set to use?

For example after a mini-batch update from the training set, what is the best thing practitioners currently do to asses how the model is doing? Specially in the context of deep learning? Should one use the same mini-batch one used to update the model or should one use the same mini-batch plus some other point from the training set. For me intuitively it seems sensible to use the mini-batch plus another fraction of the train set that isn't in the mini-batch we just used. Costing $2m$ to update.

What fraction of the Validation set to use?

This one seems a bit easier to asses what to use. Since the validation is not taking part of training I would guess that any part of it is ok to use during training. Maybe for efficiency we grab new validation set every time we get a new fraction of the data set from disk just to avoid to much disk reading? So every time we update the training set and get new data we have not trained from disk, we also update validation.

This data set will only be used to do regularization, like early stopping or choosing hyperparams, but not to actually report what the test error is (since it would be under estimating since we used it to choose the model).

What fraction of the Test set to use?

It feel this would have the same use as the test set except that we would never use it to actually choose a model, just to get a sense of the real generalization loss of our model.

If anything, one thing one can always do is that once the whole training has finished, check the actual errors (train, validation and test) on the whole data sets since at this point we are only doing it once (after the whole model has been trained).

Essentially what I am looking for is to know the data sets are used during training to evaluate the models performance.

$\endgroup$
1
$\begingroup$

Training set: simply report the loss/accuracy for every batch passed through the network while training. It's not necessary to use a different part of the training set to evaluate. While individual batches will provide a very noisy indication of how well the model is fitting, people typically smooth out the plot to give a useful signal.

Validation set: Common practice is to run validation once every fixed number of training iterations (perhaps between 1000 and 10000), on a reasonable number of validation batches (perhaps 10 or 100 or 1000). Instead of focusing on what fraction of the dataset this is, people choose based on how long it takes to run. Typically between 1% and 10% of compute time spent on validation is reasonable.

Test set: It's bad practice (if not uncommon) to evaluate on the test set while training the model or tuning parameters.

we would never use it to actually choose a model ... just to get a sense of the real generalization loss of our model.

This is easier said than done. Suppose you tweak your model a bit, and it does worse on the validation set, but actually slightly better on the test set. Do you really have the restraint to ignore this information and go back to using your worse model? Can you actually make yourself forget what parameter settings improved performance on the test set?

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.