I've a distribution of numeric scores computed using real data ( my reference distribution ). After that I generated 100000 random distributions of numeric scores computed using points picked at random in a space of possible points. So now I want to compare my reference distribtuion with these random distributions and to test whether the reference distribution is significantly different from random distributions.



I think you are concerning about level of significance or more precisely p-value. I don't know this answers your question or not, lets try with a hypothetical example.

Let us consider you are to test the probability of success of a coin. For reference distribution you are considering to be unbiased that is the probability of success=$.5$. $\hat{p}=.65$ calculated from the sample

The hypothesis to be tested

$H_0 : p=.5 \\ H_1 : p>.5$

To evaluate the problem you randomly generate 100 or 1000 data points and calculate the $\hat{p}$ under $H_0$. Calculate the p-value as follows

p-value= $P(\hat{p} > .65|H_0)$

If you are familiar with R, try the following codes

#Testing unbiasness of a coin
#H0: p = .5
#H1: p > .5 (p_hat calculated from sample=13/20)
for(i in 1:5000){
pvalue<-mean(p_hat > 13/20)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.