How different random number generators can be more similar than identical ones? I have 6 random number generators. They are "black boxes", i.e. I do not know if they are the same or different. For example I do not know if they provide the same arithmetic averages and/or root mean square deviations. My goal is to check if they are the same or different. The problem is that I can generate a limited set of numbers (every random number generator gives me only 20 numbers).
The procedure that I use is as following.


*

*I choose a pair of generators.

*I use each of them to generate 20 numbers.

*For every generator I use these numbers to calculate the mean.

*Then I calculate the difference between the means (mean1 - mean2). I will call this difference as "reference difference".


Now I want to know if this difference is "real". In other words I want to know if the observed difference in means is caused by the fact that I do have two different random number generators or I have identical generators and the mean is different just because of chance (small number of numbers).
To answer the last question I use the following procedure.


*

*I combine the two sets of numbers (20 numbers + 20 numbers).

*Then I randomly split this set into two subsets of equal size.

*For each subset I calculate means and, as before, I calculate the difference between the two means (I will call this difference as "test difference").

*I compare this test difference with the reference difference.

*I repeat this procedure many times and to see in home many cases the test difference is smaller then the reference difference.


I thought that if the reference difference is larger than the test difference in most of the cases than it is conditioned not by chance but by the fact that the compared random number generators are different. In contrast, if the generators are identical than the test difference has 50% of chances to be larger (or smaller) then the reference one.
However, I see that for the most of the considered pairs of the generators the reference difference between the averages is smaller than the test differences. How could it be? It looks like the the generators are more similar than identical generators?
 A: What you are doing is essentially the standard nonparametrical trick (a black box random generator can be seen as simply an unknown distribution, and your main hypothesis is that the distributions are the same). As such, it should work. However, you are using the difference of the means as a test statistic. Though it is not obvious to create an example, this could be very dependent on the shape of your unknown distributions.
I'm guessing that using the Wilcoxon / Mann-Whitney type of test statistic will be less influenced by weird forms of the underlying distributions (but may also be less powerful).
Regardless of the above: the idea should work (but is dependent on the fact that your samples are good representations of the true distributions). Maybe you should post some code?
As a sidenote: you do not mention any particulars about these random number generators: are they continuous (over an interval?), discrete,... ? This may also be of influence, as your sample sizes are rather small.
