I'm trying to generate i.i.d samples from two uniformly distributed random variables in MATLAB. However, when I correlate the two sets of samples, I find that the correlation is almost zero, but the p-value is above 0.05. How should I interpret this test? Is it even possible to generate i.i.d samples in real life?
>> [r,p] = corrcoef(rand(1,1000),rand(1,1000))
r =
1.0000 -0.0508
-0.0508 1.0000
p =
1.0000 0.1082
0.1082 1.0000
Recall the definition of the p value: it is the probability of observing a test statistic as extreme as the one you observed or more extreme under the null hypothesis.
Or, looking at this under a slightly different angle: If your null hypothesis holds, then your p value is a random variable that is uniformly distributed in $[0,1]$.
Try this. Re-run your code and note the p values you get. Make a histogram. You should get a flat histogram with a uniform distribution.
One consequence is that "small" and "large" p values will occur equally often under the null hypothesis. For instance, you should see a $p<0.05$ and a $p>0.95$ each about in one in twenty cases.
Another consequence is that you shouldn't expect large p values if your null hypothesis holds. What you should expect is $U[0,1]$-distributed p values. Unfortunately, this is rather hard to observe if all you have is a single realization, i.e., a single p value.
The dance of the p values is a short and very enlightening video that may be helpful.
