4
$\begingroup$

For example, when training on CIFAR10, each minibatch typically contains images from all 10 classes (assuming a moderately large batch size such as 64). What could go wrong if I train on a homogeneous batch of airplane images, followed by a homogeneous batch of automobile images, and so on?

I want to do that in order to calculate some statistics within each class during the training.

$\endgroup$

1 Answer 1

2
$\begingroup$

Stochastic gradient descent (and variants) works only with the assumption of (in poor words) representative gradients, i.e. the distribution should be almost similar (representative) of your dataset.

Pretty sure that you have no guarantees that SGD converges in that case (might happen however, but you can't know that)

If you come from Statistic, you could see this in this fashion:

SGD uses an estimator of the gradient (which variance depends on which algorithm of the SGD family you pick).
Now, this works only if the estimator is well defined

See "Curriculum Learning"

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.