Assume I have a couple of thousand hens that I want to classify into those that never lay an egg and those that will at some point in their life lay an egg. Assume that already works perfectly.
Now there are a few hens who do lay eggs, but at some point won't do so for a couple of years. Those hens are a really small minority - let's say a hundred.
Now, i want my network to classify a hen that at some point will lay an egg but won't do so for a couple of years as a third class.
My intuition tells me, if I oversample the minority category, my model will simply memorize those hundred examples and fail at generalizing.
However when using weights, my intuition would tell me that my model can't memorize those samples because it doesn't encounter them frequently enough - kind of like a higher learning rate leads to better generalisation, but worse fitting due to the coarse steps.
However all the posts on CrossValidated actually say that oversampling works better - but why? Is that also the case for really small classes like in my case?