Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite

This article uses the following form of the chi-squared test:

$\chi^2=\sum\limits_{i}\frac{(m_i-n_i)^2}{m_i+n_i}$

where $m_i$ is the number of counts of bin $i$ in a modeled histogram and $n_i$ the equivalent for the observed histogram.

  1. How does one obtain this form of the $\chi^2$ test?
  2. Is this a non-parametric statistic or am I assuming a Gaussian distribution of the data?

Edit: so, based on @Michael's answer, am I to assume that the statistics used in that article are in fact wrong?

share|improve this question

2 Answers

up vote 3 down vote accepted

1) The authors of the article have written that both $m_i$ and $n_i$ are distributed Poisson, as it's a comparison of simulated to actual data; hence the difference between the two has variance $m_i + n_i$ instead of the usual case, where $m_i$ is the variance (assuming correctness of the model). Consequently, $m_i+n_i$ is the appropriate divisor.

2) The test is indeed nonparametric. No Gaussianity assumed!

Edit: Michael Chernick makes a valuable point in comments, namely that the approximation will be poor if the cells are sparse. The authors do mention the importance of bin size selection, but it's certainly not as though you can take this statistic, apply it without care, and expect to get good results.

Edit the Second: Michael Chernick makes another valuable point in comments, which is that there are better ways of validating their model against the star data. Even if they take the "binning" approach, they'd be better off using model-calculated expected values for bin occupancy counts and doing a more typical $\chi^2$ test than comparing simulated data with actual data; the use of simulated data vs. expected values just adds randomness to the results and thereby reduces the power of the test. It may be, however, that w/o access to the software tool's code, they can't actually do the needed calculations to get bin occupancy rates, in which case this may be about as well as they can do (I could be wrong about that, though.)

share|improve this answer
Yes jbowman but the distribution is asymptotic and not exact and the approximation is bad if it has very sparse cells. The model is the assumed population distribution, The ni are not random variables they are known expect counts given the assumed Poisson distribution. I don't think the explanation you attribute to the author can hold up. – Michael Chernick May 19 '12 at 2:50
1  
Wrt "the $n_i$ are not random variables..." apparently they are, it is a comparison of synthetic data to observed data, not a model to observed data. I went back and read the entire paper, not just the section on the test, and, for example, "The whole set of synthetic populations (each with one ∼ 50 000 stars) have been simulated using the IAC-STAR ..., which generates synthetic CMDs for a given SFH and metallicity function." No disagreement about its being a poor test w/ sparse cells, but they do emphasize the importance of bin size selection, at least. I wouldn't have done it that way... – jbowman May 19 '12 at 14:07
1  
These are constants. although it is possible to compare two data sets to see if they come from the same distribution, there is no reason to artificially create a second data set from a Poisson process to do the chi square test. It is a less powerful – Michael Chernick May 19 '12 at 17:10
1  
I know that, but nonetheless that's what they did. They mention several times that they are creating synthetic data sets, eight in fact. If you don't know what I'm talking about, maybe you should read the paper in the link provided by the OP - maybe I've missed something, but maybe you have too. You seem to have leapt to a conclusion about what the problem is - testing for Poissonianity - but that isn't what the problem is. The problem is testing whether the actual star data and the simulated star data come from the same dist'n. No argument about whether this is a good approach, however! – jbowman May 19 '12 at 17:31
1  
Okay jbowman. Then is the test statistic they use asymptotically chi square and what are the degrees of freedom if so? We still haven't answered the first part of the question and we should make a point to the OP that he can do this better without the simulated data. – Michael Chernick May 19 '12 at 17:52
show 3 more comments
up vote 2 down vote

The usual chi square test would only have mi in the denominator not the sum. Chi square is an asymptotic distribution for the test statisitc and doesn't require any Gaussian assumption. I don't think the test statistic you are using is the right one and since the one with only mi in the denominator is asymptotically chi square I don't think this one is.

share|improve this answer
I assume that the reason for having the $m_i+n_i$ sum in the denominator is to avoid the issue that arises when $m_i=0$. I'm not really using it, I found it on that article. – Gabriel May 18 '12 at 13:01
1  
The practical rule of thumb (based on William Cochran's research) is that the chi square approximation goes bad with sparse cells (sparse is usually defined as less than 5). If mi is 0 then ni is also necessarily 0 if the model is right. So you would pick a wider bin that would encompass an mi>5 and hopefully ni will be greater than 5 also. Even with mi+ni in the denominator the dividing by 0 problem may not go away. – Michael Chernick May 18 '12 at 14:43
My other point was that putting mi in the denominator changes the test statistic and its asymptotic distribution. It may not even be chi square as far as I can see. – Michael Chernick May 18 '12 at 14:44
Michael, quick question. You mention Cochran's work on how the $\chi^2$ statistics is not appropriate when counts in cells are sparse (<5) Would it be correct to cite this article as the source of that claim? – Gabriel May 19 '12 at 22:30
@Gaba_p Thanks for finding that survey article. I think it is not the original source but he probably references it in that article. Honestly this is something i have heard or read about, But I have not seen the original source. Cochran obviously did a lot of work on the chi square test and the Annals of Math Stat survey article that you found probably is fascinating reading and though dated problably contains a lot of information that has been lost to the younger generations (that includes me even though I am 65). – Michael Chernick May 19 '12 at 22:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.