9
$\begingroup$

I'd like to understand the use of Monte Carlo simulation in the chisq.test() function in R.

I have a qualitative variable which has 128 levels / classes. My sample size is 26 (I was not able to sample more "individuals"). So obviously, I will have some levels with 0 "individuals". But the fact is that I have only a very small number of classes represented out of the 127 possible. As I have heard that to apply chi-squared test we should have at least 5 individuals in each level (I do not completely understand the reason for that), I thought I had to use the simulate.p.value option to use Monte Carlo simulation to estimate the distribution and compute a p-value. Without Monte Carlo simulation, R gives me a p-value < 1e-16. With Monte Carlo simulation, it gives me a p-value at 4e-5.

I tried to compute the p-value with a vector of 26 ones and 101 zeros, and with Monte-Carlo simulation, I get a p-value at 1.

Is it OK to state that, even if my sample size is small compared with the number of possible classes, the observed distribution is such that it is very unlikely that all possible classes exist at the same probability (1/127) in the real population?

$\endgroup$
  • 3
    $\begingroup$ If your data really are that you observed 26 distinct classes out of a sample of 26, then you have essentially no evidence against the hypothesis that all 127 classes have equal probability. This can be assessed with a multinomial distribution calculation. $\endgroup$ – whuber Jun 27 '13 at 13:49
  • 1
    $\begingroup$ "As I have heard that to apply chi-squared test we should have at least 5 individuals in each level (I do not completely understand the reason for that)" -- not quite. The original advice was that the expected count, not the actual count should be at least 5. The aim with that (now long outdated) rule was to try to make sure the chi-square distribution is a reasonable approximation to the discrete distribution of the test statistic. Advice across a slew of papers over the last 4 decades or so is 'that rule is somewhat too strict'. $\endgroup$ – Glen_b Aug 26 '13 at 23:58
5
$\begingroup$

By searching, it seems that the point of Monte-Carlo Simulation is to produce a reference distribution, based on randomly generated samples which will have the same size as the tested sample, in order to compute p-values when test conditions are not satisfied.

This is explained in Hope A. J Royal Stat Society Series B (1968) which can be found on JSTOR.

Here is a relevant quote from the Hope paper:

Monte-Carlo significance test procedures consist of the comparison of the observed data with random samples generated in accordance with the hypothesis being tested. ... It is preferable to use a known test of good efficiency instead of a Monte-Carlo test procedure assuming that the alternative statistical hypothesis can be completely specified. However, it is not always possible to use such a test because the necessary conditions for applying the test may not be satisfied, or the underlying distribution may be unknown or it may be difficult to decide on an appropriate test criterion.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.