I am doing chi-square tests using Scipy. Just realized the test statistics will change even though you simply rescale all data and the testing outcome could change.

For example, using chisquare([8,5,9,2,7,5], f_exp = [6,6,6,6,6,6]), it gives test statistic = 5.3333, p-value = 0.377. But if I just naively double the input data, like chisquare([16,10,18,4,14,10], f_exp = [12,12,12,12,12,12]), the test statistic = 10.6667 (of course double), and p-value=0.0584. The test statistic doubles, and p-value decreases significantly.

I understand that, from the definition of Chi-Square, the test statistic will double and p-value will change. But I think rescaling data should not change statistic testing results. What logic do I make mistake here?

  • $\begingroup$ You might look up the effect size statistics phi and Cramer's V. These won't change if you e.g. double the counts in the table. The p-value and the effect size statistics give you different kinds of information. $\endgroup$ – Sal Mangiafico Jan 10 '19 at 17:58

If you I understand your question correctly, you are not re-scaling the data, but are incorrectly claiming to have twice as much data as you do.

If you roll a six-sided die $8+5+9+2+7+5 = 36$ times to assess its fairness, then you "expect" to see each face 6 times, and the chi-squared statistic is $Q = \sum_{i=1}^6 \frac{(x_i - 6)^2}{6} \approx 5.33$ as you say. The critical value for a test at the 5% level is $c = 11.07,$ based on the chi-squared distribution with $6-1 = 5.$ degrees of freedom. So, even though your counts vary about the expected value $6,$ you do not have evidence to say the die is unfair.

However, to take your multiplication to absurd extremes, 360 rolls of the die with face counts $(80, 50, 90, 20, 70, 50)$ would give chi-squared statistic $Q \approx 53.3 >> 11.07,$ and you would not hesitate to declare the die unfair.

Notice that in a chi-squared test, the degrees of freedom depend on the number of categories, not the number of rolls of the die.

This example illustrates that a bar graph of counts can be misleading as a graphic to decide goodness-of-fit. It is a good idea to show the total number of observations in the caption; otherwise, the vertical scale is the only clue to the size of the experiment. The bar graph below shows the original counts; a bar graph for the fake experiment with 360 rolls of the die would look the same except for the numbers on the vertical scale. (In general, it is often a challenge to make useful graphs for categorical data.)

enter image description here

Note: By contrast, suppose you are doing a one-sample t test, with $n = 36$ normal observations, $\bar X = 101$ and $S = 3.2,$ and you are testing $H_0: \mu = 100$ vs. $H_a: \mu \ne 100.$ Then you will have $T = (101 - 100)\sqrt{36}/3.2 = 1.88,$ and you will not reject $H_0$ at the 5% level. If the original data were in inches and you multiply by the appropriate constant to express them in millimeters, then you still have $T = 1.88, n = 36, \nu = n-1 = 35$ and you will still not reject.

Here the degrees of freedom $\nu$ depend on the sample size, which has not changed by re-scaling the data from inches to millimeters. However, you might get a different result by using a nonlinear transformation (such as changing US miles per gallon to liters per 100 km), which would also change normal data to nonnormal data.


Just realized the test statistics will change even though you simply rescale all data

Indeed. Don't do that.

What logic do I make mistake here?

Counts don't "scale".

When you multiply a variable by a constant, the variance scales by the square: $\text{Var}(kX) = k^2 \text{Var}(X)$. This is the behavior you're anticipating. This means that the "unusualness" of an observation doesn't change by scaling it -- for example, consider a 'typical' z-score: $(X-\mu)/\sigma$ is the same as $(kX-k\mu)/(k\sigma)$

But for typical models for counts, its different; they don't have a separate variance, but rather the variance is a function of the mean. The simplest case is Poisson counts (one of the cases for which a chi-squared test is suitable after conditioning on the margins). With a Poisson, the variance is equal to the mean. If you observe this process for longer, the mean will increase in proportion but you have more information, and so the standard deviation increases more slowly.

If you double the count you're saying "I know this value relatively more precisely" (the way you get bigger counts is observing the process for longer -- gathering more data; when you double the counts you're asserting you have done so, even though you haven't)

For a Poisson variate, a 'z-score' is $(O_i-E_i)/\sqrt{E_i}$. If you double $O_i$ and $E_i$ without having more information you will compute its z-score as $(2O_i-2E_i)/\sqrt{2E_i}$ - this is $\sqrt{2}$ times as large as the z-score you started with (and the contribution to chi-squared is thereby doubled). You're incorrectly putting additional weight on this discrepancy in count. (A multinomial count has a somewhat different variance function but the underlying issue is basically the same - you can't just scale the count.)

Deal with counts as counts. Don't replace them with percentages, nor with counts in thousands. Just as counts -- so the chi-squared statistic can correctly scale them for the variance.

  • $\begingroup$ But I still have two questions. (1) What if two samples have different sample sizes? In this case how do we test the counts? (2) Back to the dice example and assuming we sample 6 million times. The frequencies go close to millions, and if difference between sample 1 and sample 2 only just few thousand? In this case the test statistic may be a big number so we reject the null hypothesis. But to me, few thousand is small compared to million times. How to give intuition for this? $\endgroup$ – Hsiang Hung Jan 10 '19 at 19:55
  • $\begingroup$ @Glen_b. Especially like your last paragraph. $\endgroup$ – BruceET Jan 10 '19 at 22:40
  • $\begingroup$ @HsiangHung: (1) Comparing 2 samples w/ each other to see if they come from populations with the same probability dist'n is a separate topic: "Chi-sq test of homogeneity." (2) Let's stick to one sample compared with one specific dist'n here. Try to understand how to formulate $H_0,$ how to compute the chi-squared test statistic, how to get the critical region, and how to decide whether to reject $H_0.$ // Work several related problems to develop some useful intuition. // Then explore how to compare two samples from separate populations with each other. Maybe ask another Q on that. $\endgroup$ – BruceET Jan 10 '19 at 22:50
  • $\begingroup$ @BruceET, thank you. I think chi-sq test of homogeneity or chi-square test of association is the answer I am looking for! $\endgroup$ – Hsiang Hung Jan 16 '19 at 18:46

Not the answer you're looking for? Browse other questions tagged or ask your own question.