I've read that the chi square test is useful to see if a sample is significantly different from a set of expected values.

For example, here is a table of results of a survey regarding people's favourite colours (n=15+13+10+17=55 total respondents):



A chi square test can tell me if this sample is significantly different from the null hypothesis of equal probability of people liking each colour.

Question: Can the test be run on the proportions of total respondents who like a certain colour? Like below:



Where, of course, 0.273+0.236+0.182+0.309=1.

If the chi square test is not suitable in this case, what test would be? Thanks!

Edit: I tried @Roman Luštrik answer below, and got the following output, why am I not getting a p-value and why does R say "Chi-squared approximation may be incorrect"?

> chisq.test(c(0,0,0,8,6,2,0,0),p = c(0.406197174,0.088746395,0.025193306,0.42041479,0.03192905,0.018328576,0.009190708,0))

    Chi-squared test for given probabilities

data:  c(0, 0, 0, 8, 6, 2, 0, 0) 
X-squared = NaN, df = 7, p-value = NA

Warning message:
In chisq.test(c(0, 0, 0, 8, 6, 2, 0, 0), p = c(0.406197174, 0.088746395,  :
  Chi-squared approximation may be incorrect
  • 1
    $\begingroup$ In the second case, are you assuming you know the total sample size? Or not? $\endgroup$
    – cardinal
    Apr 16 '11 at 22:04
  • $\begingroup$ @cardinal: yes I do know the total sample size. $\endgroup$
    – hpy
    Apr 17 '11 at 0:30
  • 3
    $\begingroup$ then just multiply the proportions by the total sample size to transform into a table of counts, and apply the chi-sq. method corresponding to your first example. $\endgroup$
    – Aaron
    Apr 17 '11 at 0:53
  • $\begingroup$ I suspect you are asking about the "goodness of fit" test (using the chi square). The use of which was explained bellow. Cheers, Tal $\endgroup$
    – Tal Galili
    Apr 18 '11 at 4:42

Correct me if I'm wrong, but I think this can be done in R using this command

> chisq.test(c(15,13,10,17))

    Chi-squared test for given probabilities

data:  c(15, 13, 10, 17) 
X-squared = 1.9455, df = 3, p-value = 0.5838

This assumes proportions of 1/4 each. You can modify expected values via argument p. For example, you think people may prefer (for whatever reason) one color over the other(s).

> chisq.test(c(15,13,10,17), p = c(0.5, 0.3, 0.1, 0.1))

    Chi-squared test for given probabilities

data:  c(15, 13, 10, 17) 
X-squared = 34.1515, df = 3, p-value = 1.841e-07
  • 2
    $\begingroup$ I suspect you're seeing this because of some low cell counts (some books I've read suggest a min. of 5 per cell). Maybe someone more knowledgeable on the subject can chip in? $\endgroup$ Apr 18 '11 at 23:48
  • 1
    $\begingroup$ Also notice that you can get a p value if you make the last of your probability more than zero (but the warning still remains). $\endgroup$ Apr 18 '11 at 23:50
  • 1
    $\begingroup$ Ott & Longnecker (An introduction to statistical methods and data analysis, 5th edition) state, on page 504, that each cell should be at least five, to use the approximation comfortably. $\endgroup$ Apr 18 '11 at 23:55
  • 1
    $\begingroup$ @penyuan : You should've mentioned that you have quite some zero counts. Roman is right, using a Chi-square in this case just doesn't work for the reasons he mentioned. $\endgroup$
    – Joris Meys
    Apr 19 '11 at 0:40
  • 1
    $\begingroup$ @penyuan : I added an answer giving you some options. $\endgroup$
    – Joris Meys
    Apr 19 '11 at 22:22

Using the extra information you gave (being that quite some of the values are 0), it's pretty obvious why your solution returns nothing. For one, you have a probability that is 0, so :

  • $e_i$ in the solution of Henry is 0 for at least one i
  • $np_i$ in the solution of probabilityislogic is 0 for at least one i

Which makes the divisions impossible. Now saying that $p=0$ means that it is impossible to have that outcome. If so, you might as well just erase it from the data (see comment of @cardinal). If you mean highly improbable, a first 'solution' might be to increase that 0 chance with a very small number.

Given :

X <- c(0,0,0,8,6,2,0,0)
p <- c(0.406197174,0.088746395,0.025193306,0.42041479,0.03192905,0.018328576,0.009190708,0)

You could do :

> p2 <- p + 1e-6
> chisq.test(X,p2)

        Pearson's Chi-squared test

data:  X and p2 
X-squared = 24, df = 21, p-value = 0.2931

But this is not a correct result. In any case, one should avoid using the chi-square test in these borderline cases. A better approach is using a bootstrap approach, calculating an adapted test statistic and comparing the one from the sample with the distribution obtained by the bootstrap.

In R code this could be (step by step) :

# The function to calculate the adapted statistic.
# We add 0.5 to the expected value to avoid dividing by 0
Statistic <- function(o,e){
    e <- e+0.5

# Set up the bootstraps, based on the multinomial distribution
n <- 10000
bootstraps <- rmultinom(n,size=sum(X),p=p)

# calculate the expected values
expected <- p*sum(X)

# calculate the statistic for the sample and the bootstrap
ChisqSamp <- Statistic(X,expected)
ChisqDist <- apply(bootstraps,2,Statistic,expected)

# calculate the p-value
p.value <- sum(ChisqSamp < sort(ChisqDist))/n

This gives a p-value of 0, which is much more in line with the difference between observed and expected. Mind you, this method assumes your data is drawn from a multinomial distribution. If this assumption doesn't hold, the p-value doesn't hold either.

  • 1
    $\begingroup$ You might reconsider your first statement, which I do not believe is correct. If $p_i = 0$ for some $i$ and the observed counts are zero (which they better be), then this just reduces to a submodel. The effect is that the number of degrees of freedom is reduced by one for each $i$ such that $p_i = 0$. For example, consider testing for uniformity of a six-sided die (that is $p_i = 1/6$ for $i \leq 6$). But, suppose we (strangely) decide to record the number of times that the numbers $1,\ldots,10$ show up. Then, the chi-square test is still valid; we just sum over the first six values. $\endgroup$
    – cardinal
    Apr 20 '11 at 12:24
  • $\begingroup$ @cardinal : I just described the data, where the expected value is 0 but the observed doesn't have to be. It's what OP gave us (although on second thought it does indeed sound rather irrealistic). Hence adding a little bit to the p value to make it highly improbable instead of impossible will help, but even then the chi-square is in this case invalid due to the large amount of table cells with counts less than 5 (as demonstrated by the code). I added the consideration in my answer, thx for the pointer. $\endgroup$
    – Joris Meys
    Apr 20 '11 at 13:10
  • $\begingroup$ yes, I'd say if $p_i = 0$, but you observe a count for that cell, then you've got more serious problems on your hands, anyways. :) $\endgroup$
    – cardinal
    Apr 20 '11 at 13:40

The chi-square test is good as long as the expected counts are large, usually above 10 is fine. below this the $\frac{1}{E(x_{i})}$ part tends to dominate the test. An exact test statistic is given by:


Where $x_{i}$ is the observed count in category $i$. $i\in \{\text{red, blue, green, yellow}\}$ in your example. $n$ is your sample size, equal to $55$ in your example. $p_i$ is the hypothesis you wish to test - the most obvious is $p_i=p_j$ (all probabilities are equal). You can show that the chi-square statistic:

$$\chi^{2}=\sum_{i}\frac{(x_{i}-np_{i})^{2}}{np_{i}}\approx 2\psi$$

In terms of the observed frequencies $f_{i}=\frac{x_{i}}{n}$ we get:

$$\psi=n\sum_{i}f_{i}\log\left(\frac{f_{i}}{p_{i}}\right)$$ $$\chi^{2}=n\sum_{i}\frac{(f_{i}-p_{i})^{2}}{p_{i}}$$

(Note that $\psi$ is the effectively the KL divergence between the hypothesis and the observed values). You may be able to see intuitively why $\psi$ is better for small $p_{i}$, because it does have a $\frac{1}{p_{i}}$ but it also has a log function which is absent from the chi-square, this "reigns in" the extreme values caused by small expected counts. Now the "exactness" of this $\psi$ statistic is not as an exact chi-square distribution - it is exact in a probability sense. The exactness comes about in the following manner, from Jaynes 2003 probability theory: the logic of science.

If you have two hypothesis $H_{1}$ and $H_{2}$ (i.e. two sets of $p_i$ values) that you wish to test, each with test statistics $\psi_{1}$ and $\psi_{2}$ respectively, then $\exp\left(\psi_{1}-\psi_{2}\right)$ gives you the likelihood ratio for $H_{2}$ over $H_{1}$. $\exp\left(\frac{1}{2}\chi_{1}^{2}-\frac{1}{2}\chi_{2}^{2}\right)$ gives an approximation to this likelihood ratio.

Now if you choose $H_{2}$ to be the "sure thing" or "perfect fit" hypothesis, then we will have $\psi_{2}=\chi^{2}_{2}=0$, and thus the chi-square and psi statistic both tell you "how far" from the perfect fit any single hypothesis is, from one which fit the observed data exactly.

Final recommendation: Use $\chi_{2}^{2}$ statistic when the expected counts are large, mainly because most statistical packages will easily report this value. If some expected counts are small, say about $np_{i}<10$, then use $\psi$, because the chi-square is a bad approximation in this case, these small cells will dominate the chi-square statistic.

  • 1
    $\begingroup$ I'm pretty sure the expected frequencies can't be larger than 10. :) $\endgroup$
    – cardinal
    Apr 17 '11 at 0:59
  • $\begingroup$ @cardinal - glad that this was your objection - for it means the rest of my answer must have been good :). $\endgroup$ Apr 17 '11 at 1:04
  • $\begingroup$ Wow, I hope I'm not getting a reputation for being so picky/grumpy. $\endgroup$
    – cardinal
    Apr 17 '11 at 1:46
  • 1
    $\begingroup$ I'm don't quite follow the "exact" terminology. Perhaps that is particular to Jaynes' work. Your $\psi$ is the log-likelihood-ratio test statistic though and so $2 \psi$ is asymptotically distributed as a $\chi^2$ distribution by Wilks' theorem. Also, $\chi^2 - 2 \psi \to 0$ in probability, which by Slutsky's theorem is enough to conclude that $\chi^2$ has the same distribution as $2\psi$. Finally, it turns out that $\chi^2$ is the scorte test statistic in this problem as well, which provides another connection between the two test statistics. $\endgroup$
    – cardinal
    Apr 17 '11 at 1:53
  • $\begingroup$ Also, Agresti (Categorical Data Analysis, 2nd ed., p. 80) claims that $\chi^2$ actually converges to a chi-squared distribution faster than $2 \psi$, which seems at odds with your recommendation. :) $\endgroup$
    – cardinal
    Apr 17 '11 at 1:55

Yes, you can test the null hypothesis: "H0: prop(red)=prop(blue)=prop(green)=prop(yellow)=1/4" using a chi square test that compares the proportions of the survey (0.273, ...) to the expected proportions (1/4, 1/4, 1/4, 1/4)

  • $\begingroup$ Just to confirm, it will also work with expected proportions that are unequal to each other? $\endgroup$
    – hpy
    Apr 17 '11 at 0:31
  • 4
    $\begingroup$ the test won't be meaningful unless you know the full sample size. Proportions of 1.0 / 0.0 / 0.0 / 0.0 mean very different things if they are from a sample of size 1 as opposed a sample of size 100. $\endgroup$
    – Aaron
    Apr 17 '11 at 0:52
  • $\begingroup$ Yes, I DO know the total sample size. $\endgroup$
    – hpy
    Apr 17 '11 at 14:08

The test statistic for Pearson's chi-square test is

$$\sum_{i=1}^{n} \frac{(O_i - E_i)^2}{E_i}$$

If you write $o_i = \dfrac{O_i}{n}$ and $e_i = \dfrac{E_i}{n}$ to have proportions, where $n=\sum_{i=1}^{n} O_i$ is the sample size and $\sum_{i=1}^{n} e_i =1$, then the test statistic is is equal to

$$n \sum_{i=1}^{n} \frac{(o_i - e_i)^2}{e_i}$$

so a test of the significance of the observed proportions depends on the sample size, much as one would expect.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.