8
$\begingroup$

I'm working on developing a physics lab about radioactive decay, and in analyzing sample data I've taken, I ran into a statistics issue that surprised me.

It is well known that the number of decays per unit time by a radioactive source is Poisson distributed. The way the lab works is that students count the number of decays per time window, and then repeat this many many times. Then they bin their data by the number of counts, and do a $\chi^2$ goodness of fit test with 1 parameter estimated (the mean) to check whether or not the null hypothesis (the data is drawn from a Poisson distribution with the estimated mean value) holds. Hopefully they'll get a large p-value and conclude that physics indeed works (yay).

I noticed that the way I binned my data had a large effect on the p-value. For example, if I chose lots of very small bins (e.g. a separate bin for each integer: 78 counts/min, 79 counts/min, etc.) I got a small p-value, and would have had to reject the null hypothesis. If, however, I binned my data into fewer bins (e.g. using the number of bins given by Sturge's Rule: $1+log_{2}(N)$), I got a much larger p-value, and did NOT reject the null hypothesis.

Looking at my data, it looks extremely Poisson-distributed (It lines up almost perfectly with my expected counts/minutes). That said, there are a few counts in bins very far away from the mean. That means when computing the $\chi^2$ statistic using very small bins, I have a few terms like: $$\frac{(Observed-Expected)^2}{Expected} = \frac{(1-0.05)^2}{0.05}=18.05$$ This leads to a high $\chi^2$ statistic, and thus a low p-value. As expected, the problem goes away for larger bin widths, since the expected value never gets that low.

Questions:

Is there a good rule of thumb for choosing bin sizes when doing a $\chi^2$ GOF test?

Is this discrepancy between outcomes for different bin sizes something that I should have known about*, or is indicative of some larger problem in my proposed data analysis?

- Thank you

*(I took a stats class in undergrad, but it's not my area of expertise.)

Improve this question
$\endgroup$
15
  • $\begingroup$ Seems like a sensitivity and specificity issue, i.e. you're getting type-II errors because you're measurements are too specific. $\endgroup$
    – Jay Schyler Raadt
    Dec 23, 2017 at 21:52
  • 1
    $\begingroup$ A measurement that is too specific will produce type-II errors, but one that is too sensitive will produce type-I errors. For example, a very specific cutoff for an IQ test could leave a child with an IQ of 70.1 not qualifying for special education whereas a child with an IQ of 69.9 does. This would be a type-II error, where the null hypothesis "this child does not qualify" is falsely not rejected. Thus, a more sensitive measurement is needed, a bigger net, although too big a net might cause a type-I error where the null hypothesis is falsely rejected. $\endgroup$
    – Jay Schyler Raadt
    Dec 23, 2017 at 22:03
  • 2
    $\begingroup$ 1. The chi-square approximation can be quite poor if you have small expected values -- but you don't have to have constant bin-width either (as long as you're not choosing it with reference to the values of the observed counts). 2. "Hopefully they'll get a large p-value and conclude that physics indeed works (yay)." -- I expect you already know, but it should be made clear: failure to reject the null doesn't confirm that the null is true; it suggests that any deviation from Poisson wasn't large enough to reliably detect. ... ctd $\endgroup$
    – Glen_b
    Dec 24, 2017 at 9:36
  • 1
    $\begingroup$ OK, thank you all for your attention to this. @Whuber, your answer to the other question is incredible. Would you, then, say that the answer to my first question is basically just, "no" -- there is no good rule of thumb at this level? $\endgroup$
    – Bunji
    Apr 23, 2019 at 20:19
  • 1
    $\begingroup$ There are many considerations. I think there may be some useful rules of thumb. For instance, I have usually been successful by guessing what the distribution of counts will be and creating bins expected to have approximately equal counts of 5 or more each; but it's rare to need more than 20 bins. Sometimes I'm looking for discrepancies within particular ranges, such as the distributional tails, and so within those ranges I might create narrower bins in order to detect detailed differences. $\endgroup$
    – whuber ♦
    Apr 23, 2019 at 21:14

1 Answer 1

Reset to default
1
$\begingroup$

Is this discrepancy between outcomes for different bin sizes something that I should have known about*, or is indicative of some larger problem in my proposed data analysis?

The binning of the radioactive decay sample set is a red herring here. The real problem originates from the fact that chi-square (alongside other hypothesis testing frameworks) is highly sensitive to sample size. In the case of chi-square, as sample size increases, absolute differences become an increasingly smaller portion of the expected value. As such, if the sample size is very large we may find small p-values and statistical significance when the findings are small and uninteresting. Conversely, a reasonably strong association may not come up as significant if the sample size is small.

Is there a good rule of thumb for choosing bin sizes when doing a χ2 GOF test?

The answer seems that one should not aim to find the right N (I am not sure it is doable, but would be great if someone else chips in to contradict), but look beyond p-values solely when N is high. This seems a good paper on the subject: Too Big to Fail: Large Samples and the p-Value Problem

P.S. There are alternatives to χ2 test such as Cramer's V and G-Test; however you will still hit the same issues with large N -> small p-value.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.