2
$\begingroup$

I have a computer model simulation of a real world machine that generates either blue or red colours as output (as an example). Suppose the expected proportion of blue is 0.2 and expected red is 0.8 in the real world (from the real world machine) and the computer model generated proportions are 0.3 blue and 0.7 red, my understanding is that I can use the Chi-square test to check if the computer output matches the real world data (null hypothesis being there is no difference between the computer generated output and the real world output).

However, let's say I introduce random noise into the computer simulation and that I can change the random seed 200 times to give rise to 200 different sets of results. Instead of doing Chi-square test 200 times on each individual simulation, what statistical analysis would be appropriate to check the null hypothesis?

In my search, I have found suggestions such as the Mcnemar test but I believe none quite fits my specific case.

$\endgroup$
2
  • $\begingroup$ Is it reasonable in your setup to assume that the random noise in the computer simulation will not change the expected proportions of blue/red from one realization to another? $\endgroup$
    – Ben Bolker
    Commented Jul 9, 2023 at 16:36
  • $\begingroup$ @BenBolker if I understand you correctly, yes there is no change to the expected proportion (i.e. proportion generated in the real world machine) with each simulation with different random seed. Ultimately, the expected proportion is like ground truth, and I am trying to verify the computer simulation's veracity in modelling this real world machine. $\endgroup$
    – sunnydk
    Commented Jul 9, 2023 at 16:43

1 Answer 1

6
$\begingroup$

If you are willing to assume that the expected proportion should be the same in each simulation (and that you're not, for example, varying the expected proportion around the overall expected mean from one simulation to the next), then you should be able to pool the results from all of the simulations into one big chi-squared test. For example, if your results from three successive simulations are [(28, 72), (35, 65), (27, 73)], then you can do a chi-squared test with [(28+35+27), (72+65+73)] versus expected values (300*0.2, 300*0.8) (with one degree of freedom, since you're using the total number of counts from the simulations to compute the expected values).

The reason this works is that the only assumption of the chi-squared test is that the counts are independent of each other and homogeneous, which is still true if you combine values across independent simulations.

Note that you must compute the overall observed proportion as though you had combined all of the observations into one big data set ($\sum b_i/\sum(b_i + r_i)$), not as the mean of the proportions observed in each simulation ($\sum(b_i/r_i)/n$): these are different, e.g.

> b <- c(28, 35, 27)
> (r <- 100-b)
[1] 72 65 73
> sum(b)/sum(r)
[1] 0.4285714
> mean(b/(b+r))
[1] 0.3
$\endgroup$
4
  • $\begingroup$ Thank you so much! Just one follow up question: can I also just take the average proportion for each colour across the 200 simulations to perform a basic Chi-square test. Will this be the same as your suggestion? $\endgroup$
    – sunnydk
    Commented Jul 9, 2023 at 16:58
  • $\begingroup$ Not sure what you mean. Chi-square must be done on counts, and the counts must be totaled as discussed in my edit. $\endgroup$
    – Ben Bolker
    Commented Jul 9, 2023 at 17:15
  • $\begingroup$ @BenBolker. I thought the chi-square distribution arose as the sum of squared Normal distributions? $\endgroup$
    – DWin
    Commented Jul 10, 2023 at 15:02
  • $\begingroup$ The chi-squared distribution is indeed the sum of squared standard Normals, but in the case of Pearson's X^2 test those Normals are themselves approximations of Poisson counts ... (expected-observed)^2/expected is the squared deviation, normalized by the expected variance $\endgroup$
    – Ben Bolker
    Commented Jul 10, 2023 at 19:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.