14
$\begingroup$

I've read that the chi square test is useful to see if a sample is significantly different from a set of expected values.

For example, here is a table of results of a survey regarding people's favourite colours (n=15+13+10+17=55 total respondents):

red, blue, green, yellow

15, 13, 10, 17

A chi square test can tell me if this sample is significantly different from the null hypothesis of equal probability of people liking each colour.

Question: Can the test be run on the proportions of total respondents who like a certain colour? Like below:

red, blue, green, yellow

0.273, 0.236, 0.182, 0.309

Where, of course, $0.273 + 0.236 + 0.182 + 0.309=1$.

If the chi square test is not suitable in this case, what test would be?

Edit: I tried @Roman Luštrik answer below, and got the following output, why am I not getting a p-value and why does R say "Chi-squared approximation may be incorrect"?

chisq.test(c(0, 0, 0, 8, 6, 2, 0, 0), p = c(0.406197174, 0.088746395, 
             0.025193306, 0.42041479, 0.03192905, 0.018328576, 
             0.009190708, 0))
    
        Chi-squared test for given probabilities
    
    data:  c(0, 0, 0, 8, 6, 2, 0, 0) 
    X-squared = NaN, df = 7, p-value = NA
    
    Warning message:
    In chisq.test(c(0, 0, 0, 8, 6, 2, 0, 0), p = c(0.406197174, 
           0.088746395,  :
      Chi-squared approximation may be incorrect  
$\endgroup$
4
  • 1
    $\begingroup$ In the second case, are you assuming you know the total sample size? Or not? $\endgroup$
    – cardinal
    Apr 16, 2011 at 22:04
  • 1
    $\begingroup$ @cardinal: yes I do know the total sample size. $\endgroup$
    – hpy
    Apr 17, 2011 at 0:30
  • 4
    $\begingroup$ then just multiply the proportions by the total sample size to transform into a table of counts, and apply the chi-sq. method corresponding to your first example. $\endgroup$
    – Aaron
    Apr 17, 2011 at 0:53
  • $\begingroup$ I suspect you are asking about the "goodness of fit" test (using the chi square). The use of which was explained bellow. Cheers, Tal $\endgroup$
    – Tal Galili
    Apr 18, 2011 at 4:42

6 Answers 6

12
$\begingroup$

Correct me if I'm wrong, but I think this can be done in R using this command

chisq.test(c(15, 13, 10, 17))
    
        Chi-squared test for given probabilities
    
    data:  c(15, 13, 10, 17) 
    X-squared = 1.9455, df = 3, p-value = 0.5838

This assumes proportions of 1/4 each. You can modify expected values via argument p. For example, you think people may prefer (for whatever reason) one color over the other(s).

chisq.test(c(15, 13, 10, 17), p = c(0.5, 0.3, 0.1, 0.1))
    
        Chi-squared test for given probabilities
    
    data:  c(15, 13, 10, 17) 
    X-squared = 34.1515, df = 3, p-value = 1.841e-07
$\endgroup$
7
  • 2
    $\begingroup$ I suspect you're seeing this because of some low cell counts (some books I've read suggest a min. of 5 per cell). Maybe someone more knowledgeable on the subject can chip in? $\endgroup$ Apr 18, 2011 at 23:48
  • 1
    $\begingroup$ Also notice that you can get a p value if you make the last of your probability more than zero (but the warning still remains). $\endgroup$ Apr 18, 2011 at 23:50
  • 1
    $\begingroup$ Ott & Longnecker (An introduction to statistical methods and data analysis, 5th edition) state, on page 504, that each cell should be at least five, to use the approximation comfortably. $\endgroup$ Apr 18, 2011 at 23:55
  • 1
    $\begingroup$ @penyuan : You should've mentioned that you have quite some zero counts. Roman is right, using a Chi-square in this case just doesn't work for the reasons he mentioned. $\endgroup$
    – Joris Meys
    Apr 19, 2011 at 0:40
  • 1
    $\begingroup$ @penyuan : I added an answer giving you some options. $\endgroup$
    – Joris Meys
    Apr 19, 2011 at 22:22
11
$\begingroup$

Using the extra information you gave (being that quite some of the values are 0), it's pretty obvious why your solution returns nothing. For one, you have a probability that is 0, so :

  • $e_i$ in the solution of Henry is 0 for at least one i
  • $np_i$ in the solution of @probabilityislogic is 0 for at least one i

Which makes the divisions impossible. Now saying that $p=0$ means that it is impossible to have that outcome. If so, you might as well just erase it from the data (see comment of @cardinal). If you mean highly improbable, a first 'solution' might be to increase that 0 chance with a very small number.

Given :

    X <- c(0, 0, 0, 8, 6, 2, 0, 0)
    p <- c(0.406197174, 0.088746395, 0.025193306, 0.42041479, 
           0.03192905, 0.018328576, 0.009190708, 0)

You could do :

p2 <- p + 1e-6
chisq.test(X, p2)
    
            Pearson's Chi-squared test
    
    data:  X and p2 
    X-squared = 24, df = 21, p-value = 0.2931

But this is not a correct result. In any case, one should avoid using the chi-square test in these borderline cases. A better approach is using a bootstrap approach, calculating an adapted test statistic and comparing the one from the sample with the distribution obtained by the bootstrap.

In R code this could be (step by step) :

    # The function to calculate the adapted statistic.
    # We add 0.5 to the expected value to avoid dividing by 0
    Statistic <- function(o,e){
        e <- e+0.5
        sum(((o-e)^2)/e)
    }
    
    # Set up the bootstraps, based on the multinomial distribution
    n <- 10000
    bootstraps <- rmultinom(n, size=sum(X), p=p)
    
    # calculate the expected values
    expected <- p*sum(X)
    
    # calculate the statistic for the sample and the bootstrap
    ChisqSamp <- Statistic(X, expected)
    ChisqDist <- apply(bootstraps, 2, Statistic, expected)
    
    # calculate the p-value
    p.value <- sum(ChisqSamp < sort(ChisqDist))/n
    p.value

This gives a p-value of 0, which is much more in line with the difference between observed and expected. Mind you, this method assumes your data is drawn from a multinomial distribution. If this assumption doesn't hold, the p-value doesn't hold either.

$\endgroup$
3
  • 1
    $\begingroup$ You might reconsider your first statement, which I do not believe is correct. If $p_i = 0$ for some $i$ and the observed counts are zero (which they better be), then this just reduces to a submodel. The effect is that the number of degrees of freedom is reduced by one for each $i$ such that $p_i = 0$. For example, consider testing for uniformity of a six-sided die (that is $p_i = 1/6$ for $i \leq 6$). But, suppose we (strangely) decide to record the number of times that the numbers $1,\ldots,10$ show up. Then, the chi-square test is still valid; we just sum over the first six values. $\endgroup$
    – cardinal
    Apr 20, 2011 at 12:24
  • $\begingroup$ @cardinal : I just described the data, where the expected value is 0 but the observed doesn't have to be. It's what OP gave us (although on second thought it does indeed sound rather irrealistic). Hence adding a little bit to the p value to make it highly improbable instead of impossible will help, but even then the chi-square is in this case invalid due to the large amount of table cells with counts less than 5 (as demonstrated by the code). I added the consideration in my answer, thx for the pointer. $\endgroup$
    – Joris Meys
    Apr 20, 2011 at 13:10
  • 1
    $\begingroup$ yes, I'd say if $p_i = 0$, but you observe a count for that cell, then you've got more serious problems on your hands, anyways. :) $\endgroup$
    – cardinal
    Apr 20, 2011 at 13:40
9
$\begingroup$

The chi-square test is good as long as the expected counts are large, usually above 10 is fine. below this the $\frac{1}{E(x_{i})}$ part tends to dominate the test. An exact test statistic is given by:

$$\psi=\sum_{i}x_{i}\log\left(\frac{x_{i}}{np_{i}}\right)$$

Where $x_{i}$ is the observed count in category $i$. $i\in \{\text{red, blue, green, yellow}\}$ in your example. $n$ is your sample size, equal to $55$ in your example. $p_i$ is the hypothesis you wish to test - the most obvious is $p_i=p_j$ (all probabilities are equal). You can show that the chi-square statistic:

$$\chi^{2}=\sum_{i}\frac{(x_{i}-np_{i})^{2}}{np_{i}}\approx 2\psi$$

In terms of the observed frequencies $f_{i}=\frac{x_{i}}{n}$ we get:

$$\psi=n\sum_{i}f_{i}\log\left(\frac{f_{i}}{p_{i}}\right)$$ $$\chi^{2}=n\sum_{i}\frac{(f_{i}-p_{i})^{2}}{p_{i}}$$

(Note that $\psi$ is the effectively the KL divergence between the hypothesis and the observed values). You may be able to see intuitively why $\psi$ is better for small $p_{i}$, because it does have a $\frac{1}{p_{i}}$ but it also has a log function which is absent from the chi-square, this "reigns in" the extreme values caused by small expected counts. Now the "exactness" of this $\psi$ statistic is not as an exact chi-square distribution - it is exact in a probability sense. The exactness comes about in the following manner, from Jaynes 2003 probability theory: the logic of science.

If you have two hypothesis $H_{1}$ and $H_{2}$ (i.e. two sets of $p_i$ values) that you wish to test, each with test statistics $\psi_{1}$ and $\psi_{2}$ respectively, then $\exp\left(\psi_{1}-\psi_{2}\right)$ gives you the likelihood ratio for $H_{2}$ over $H_{1}$. $\exp\left(\frac{1}{2}\chi_{1}^{2}-\frac{1}{2}\chi_{2}^{2}\right)$ gives an approximation to this likelihood ratio.

Now if you choose $H_{2}$ to be the "sure thing" or "perfect fit" hypothesis, then we will have $\psi_{2}=\chi^{2}_{2}=0$, and thus the chi-square and psi statistic both tell you "how far" from the perfect fit any single hypothesis is, from one which fit the observed data exactly.

Final recommendation: Use $\chi_{2}^{2}$ statistic when the expected counts are large, mainly because most statistical packages will easily report this value. If some expected counts are small, say about $np_{i}<10$, then use $\psi$, because the chi-square is a bad approximation in this case, these small cells will dominate the chi-square statistic.

$\endgroup$
27
  • 1
    $\begingroup$ I'm don't quite follow the "exact" terminology. Perhaps that is particular to Jaynes' work. Your $\psi$ is the log-likelihood-ratio test statistic though and so $2 \psi$ is asymptotically distributed as a $\chi^2$ distribution by Wilks' theorem. Also, $\chi^2 - 2 \psi \to 0$ in probability, which by Slutsky's theorem is enough to conclude that $\chi^2$ has the same distribution as $2\psi$. Finally, it turns out that $\chi^2$ is the scorte test statistic in this problem as well, which provides another connection between the two test statistics. $\endgroup$
    – cardinal
    Apr 17, 2011 at 1:53
  • $\begingroup$ Also, Agresti (Categorical Data Analysis, 2nd ed., p. 80) claims that $\chi^2$ actually converges to a chi-squared distribution faster than $2 \psi$, which seems at odds with your recommendation. :) $\endgroup$
    – cardinal
    Apr 17, 2011 at 1:55
  • 1
    $\begingroup$ @cardinal - you are focused on the distribution of the statistic. What I am saying is that the likelihood ratio is already exact using $\psi$, you can compare hypothesis directly using it, rather than a p-value based on its distribution. So if $\psi=ln(2)$ then this means that the "perfect fit" is twice as likely as the hypothesised fit - and I just realised my odds ratio is the "wrong way around" $\endgroup$ Apr 17, 2011 at 2:03
  • $\begingroup$ I didn't quite follow that last comment, but I think the odds ratio you use is correct. $\endgroup$
    – cardinal
    Apr 17, 2011 at 2:07
  • $\begingroup$ @cardinal - they have the same asymptotic distribution, but different finite sample distributions, which is why I said use chi-square when sample size is large, and $\psi$ when sample size is small. can show by easy example that $\psi$ is way better that $\chi^2$ in small expected count case. $\endgroup$ Apr 17, 2011 at 2:08
7
$\begingroup$

P value will vary with total size of sample even if proportion remains the same. This can be seen in following example with OP's proportions and varying sample size:

from statsmodels.stats.proportion import proportions_chisquare

countlist = [273, 236, 182, 309]
nobslist = [1000,1000,1000,1000]
res = proportions_chisquare(countlist, nobslist)
print("P=",res[1])

countlist = [27.3, 23.6, 18.2, 30.9]
nobslist = [100,100,100,100]
res = proportions_chisquare(countlist, nobslist)
print("P=",res[1])

countlist = [2.73, 2.36, 1.82, 3.09]
nobslist = [10,10,10,10]
res = proportions_chisquare(countlist, nobslist)
print("P=",res[1])

countlist = [.273, .236, .182, .309]
nobslist = [1,1,1,1]
res = proportions_chisquare(countlist, nobslist)
print("P=",res[1])

Four P values printed out by above code are all different:

P= 3.3202983952938086e-10
P= 0.19436113917526665
P= 0.9252292201159897
P= 0.9973200264790189

Only first of above is significant (P<0.05).

$\endgroup$
6
$\begingroup$

The test statistic for Pearson's chi-square test is

$$\sum_{i=1}^{n} \frac{(O_i - E_i)^2}{E_i}$$

If you write $o_i = \dfrac{O_i}{n}$ and $e_i = \dfrac{E_i}{n}$ to have proportions, where $n=\sum_{i=1}^{n} O_i$ is the sample size and $\sum_{i=1}^{n} e_i =1$, then the test statistic is is equal to

$$n \sum_{i=1}^{n} \frac{(o_i - e_i)^2}{e_i}$$

so a test of the significance of the observed proportions depends on the sample size, much as one would expect.

$\endgroup$
3
$\begingroup$

Yes, you can test the null hypothesis:

"H0: prop(red)=prop(blue)=prop(green)=prop(yellow)=1/4"

using a chi square test that compares the proportions of the survey (0.273, ...) to the expected proportions (1/4, 1/4, 1/4, 1/4)

$\endgroup$
3
  • $\begingroup$ Just to confirm, it will also work with expected proportions that are unequal to each other? $\endgroup$
    – hpy
    Apr 17, 2011 at 0:31
  • 4
    $\begingroup$ the test won't be meaningful unless you know the full sample size. Proportions of 1.0 / 0.0 / 0.0 / 0.0 mean very different things if they are from a sample of size 1 as opposed a sample of size 100. $\endgroup$
    – Aaron
    Apr 17, 2011 at 0:52
  • $\begingroup$ Yes, I DO know the total sample size. $\endgroup$
    – hpy
    Apr 17, 2011 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.