3
$\begingroup$

I have 6 random number generators. They are "black boxes", i.e. I do not know if they are the same or different. For example I do not know if they provide the same arithmetic averages and/or root mean square deviations. My goal is to check if they are the same or different. The problem is that I can generate a limited set of numbers (every random number generator gives me only 20 numbers).

The procedure that I use is as following.

  1. I choose a pair of generators.
  2. I use each of them to generate 20 numbers.
  3. For every generator I use these numbers to calculate the mean.
  4. Then I calculate the difference between the means (mean1 - mean2). I will call this difference as "reference difference".

Now I want to know if this difference is "real". In other words I want to know if the observed difference in means is caused by the fact that I do have two different random number generators or I have identical generators and the mean is different just because of chance (small number of numbers).

To answer the last question I use the following procedure.

  1. I combine the two sets of numbers (20 numbers + 20 numbers).
  2. Then I randomly split this set into two subsets of equal size.
  3. For each subset I calculate means and, as before, I calculate the difference between the two means (I will call this difference as "test difference").
  4. I compare this test difference with the reference difference.
  5. I repeat this procedure many times and to see in home many cases the test difference is smaller then the reference difference.

I thought that if the reference difference is larger than the test difference in most of the cases than it is conditioned not by chance but by the fact that the compared random number generators are different. In contrast, if the generators are identical than the test difference has 50% of chances to be larger (or smaller) then the reference one.

However, I see that for the most of the considered pairs of the generators the reference difference between the averages is smaller than the test differences. How could it be? It looks like the the generators are more similar than identical generators?

Improve this question
$\endgroup$
5
  • $\begingroup$ The widely used random number generators today are of high quality that will not allow to distinguish between them using small samples. $\endgroup$
    – GaBorgulya
    Commented Apr 28, 2011 at 10:55
  • $\begingroup$ Do you really mean "random number generators", i.e. numerical algorithms that aim to give observations with a known distribution (usually uniform(0,1)), or just "random variables", i.e. sources of data with unknown distributions. $\endgroup$
    – Aniko
    Commented Apr 28, 2011 at 16:11
  • $\begingroup$ Also see stats.stackexchange.com/questions/30/…, stats.stackexchange.com/questions/40/… $\endgroup$
    – GaBorgulya
    Commented Apr 28, 2011 at 22:32
  • $\begingroup$ @Aniko, you are right, it is better to say that I have a random variable. I just wanted to abstract from the details. In my case the real numbers are created by different person (one "random number generator" in my case is a human). $\endgroup$
    – Roman
    Commented Apr 29, 2011 at 7:59
  • $\begingroup$ When you say "reference differences" are smaller than the "test differences", do you mean that as in -10 is smaller than a bunch of numbers in the -5 to 5 range, or do you mean that in absolute value? The latter would imply that half are below and half are above, because the "test differences" should be symmetrically distributed around 0 (without an absolute value). $\endgroup$
    – Aniko
    Commented Apr 29, 2011 at 13:30

1 Answer 1

Reset to default
2
$\begingroup$

What you are doing is essentially the standard nonparametrical trick (a black box random generator can be seen as simply an unknown distribution, and your main hypothesis is that the distributions are the same). As such, it should work. However, you are using the difference of the means as a test statistic. Though it is not obvious to create an example, this could be very dependent on the shape of your unknown distributions.

I'm guessing that using the Wilcoxon / Mann-Whitney type of test statistic will be less influenced by weird forms of the underlying distributions (but may also be less powerful).

Regardless of the above: the idea should work (but is dependent on the fact that your samples are good representations of the true distributions). Maybe you should post some code?

As a sidenote: you do not mention any particulars about these random number generators: are they continuous (over an interval?), discrete,... ? This may also be of influence, as your sample sizes are rather small.

Improve this answer
$\endgroup$
2
  • $\begingroup$ My random number generators generate real numbers in the range from -180 to 180. I understand that equal means do not mean that distributions are the same. But if the means are different, then the distributions are also different. So, it is why I want to check the means. My problem is that the difference between means is smaller than the difference caused by chance. I cannot understand how it is possible. $\endgroup$
    – Roman
    Commented Apr 28, 2011 at 12:23
  • $\begingroup$ Like I said: I agree that comparing two distributions should certainly not give less different means than identical ones (at least not consistently so). Post some code and/or example data, and maybe we can help pick up what went wrong? Note that when the means are the same, the distribution could still be very different (think of a bimodal distribution centered around the mean of a standard normal one, or a more skewed distribution). $\endgroup$
    – Nick Sabbe
    Commented Apr 28, 2011 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.