It is common in Deep Learning to use SGD (with mini batches). However it is also common that the data sets are too large to compute the full current accuracy of the model using the whole data set as it trains. Thus, it seems that one alternative is to instead use fractions of the data set to asses how the training is going as SGD trains the model. It seems the reasons one would do this is because checking for every update on the full data set how the model is doing is ridiculously expensive in time $O(NT)$ where $N$ is the size of the data set and $T$ is the update iterations. It also seems unnecessary because as the $N$ increases it seems we only get sub linear gains on the accuracy. I base that because the standard dev of sample mean grows as $\frac{\sigma}{ \sqrt n}$ and not $\frac{\sigma}{ n }$, so for every doubling of the data set size we only gain an increased accuracy of $\sqrt 2$. So it takes twice as long to compute but we don't get an estimate twice as accurate but less.
Thus my question is about the best practices to asses how the model is doing using fractions of the train, validation and test set (specially in the context of deep learning).
What fraction of the Training set to use?
For example after a mini-batch update from the training set, what is the best thing practitioners currently do to asses how the model is doing? Specially in the context of deep learning? Should one use the same mini-batch one used to update the model or should one use the same mini-batch plus some other point from the training set. For me intuitively it seems sensible to use the mini-batch plus another fraction of the train set that isn't in the mini-batch we just used. Costing $2m$ to update.
What fraction of the Validation set to use?
This one seems a bit easier to asses what to use. Since the validation is not taking part of training I would guess that any part of it is ok to use during training. Maybe for efficiency we grab new validation set every time we get a new fraction of the data set from disk just to avoid to much disk reading? So every time we update the training set and get new data we have not trained from disk, we also update validation.
This data set will only be used to do regularization, like early stopping or choosing hyperparams, but not to actually report what the test error is (since it would be under estimating since we used it to choose the model).
What fraction of the Test set to use?
It feel this would have the same use as the test set except that we would never use it to actually choose a model, just to get a sense of the real generalization loss of our model.
If anything, one thing one can always do is that once the whole training has finished, check the actual errors (train, validation and test) on the whole data sets since at this point we are only doing it once (after the whole model has been trained).
Essentially what I am looking for is to know the data sets are used during training to evaluate the models performance.
