I have a data set with $C=64$ classes and $N_c = 600$ images and total images $N = \sum_c N_c$. Each class has a separate folder for each class. I want to be able to sample images as if they were in a union of all folders (i.e. any image has the same probability no matter where it is located and even if classes are unbalanced). The issue is that I don't want to actually flatten the data set into a union as that duplicates the data and higher storage costs. So I want to sample such that if I were to sample a batch e.g. $B=1024$ then every image has the same chance of occurring. I think for that I need that the probability of an image to be equal to:

$$ Pr[X = x] = \frac{1}{N}$$

I think that is what my problem boils down to if I am correct (correct me if that is the wrong modeling please!)

So for that to happen I think what I need to do is first sample a class with probability equal $$Pr[C=c] = \frac{N_c C}{N}$$ and sampling an image given that class uniformly i.e. with probability $$ Pr[X=x \mid C = c] = \frac{1}{N_c}$$

This is what I get using the marginalization rule for probs:

$$ Pr[X=x] = \sum_c p(c) p(x \mid c) = \frac{1}{N} $$

if you plug in the numbers.

But that seems weird, I would have thought that it's sufficient to simply sample any class weighted by the number of images it has and then sample a image uniformly (i.e. just sample uniformly twice $B$ times to get your batch).

But formal calculations seem to give me something else, what is my mistake?


just create list of all files, then sample this list and voila!,

as for calculations:

probability sampling image from class $A$ sampling from whole set is $P(class\_A)=size(A)/size(whole\_set)$ but in second sampling/drawing $P(class\_A)=\frac{size(A)-1}{size(whole\_set)-1}$ because I presume you are sampling without replacement, of course after picking class/subset you can sample uniformly from this smaller set


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.