I have 6 random number generators. They are "black boxes", i.e. I do not know if they are the same or different. For example I do not know if they provide the same arithmetic averages and/or root mean square deviations. My goal is to check if they are the same or different. The problem is that I can generate a limited set of numbers (every random number generator gives me only 20 numbers).
The procedure that I use is as following.
- I choose a pair of generators.
- I use each of them to generate 20 numbers.
- For every generator I use these numbers to calculate the mean.
- Then I calculate the difference between the means (mean1 - mean2). I will call this difference as "reference difference".
Now I want to know if this difference is "real". In other words I want to know if the observed difference in means is caused by the fact that I do have two different random number generators or I have identical generators and the mean is different just because of chance (small number of numbers).
To answer the last question I use the following procedure.
- I combine the two sets of numbers (20 numbers + 20 numbers).
- Then I randomly split this set into two subsets of equal size.
- For each subset I calculate means and, as before, I calculate the difference between the two means (I will call this difference as "test difference").
- I compare this test difference with the reference difference.
- I repeat this procedure many times and to see in home many cases the test difference is smaller then the reference difference.
I thought that if the reference difference is larger than the test difference in most of the cases than it is conditioned not by chance but by the fact that the compared random number generators are different. In contrast, if the generators are identical than the test difference has 50% of chances to be larger (or smaller) then the reference one.
However, I see that for the most of the considered pairs of the generators the reference difference between the averages is smaller than the test differences. How could it be? It looks like the the generators are more similar than identical generators?