# Can chi square be used to compare proportions?

I've read that the chi square test is useful to see if a sample is significantly different from a set of expected values.

For example, here is a table of results of a survey regarding people's favourite colours (n=15+13+10+17=55 total respondents):

red,blue,green,yellow

15,13,10,17


A chi square test can tell me if this sample is significantly different from the null hypothesis of equal probability of people liking each colour.

Question: Can the test be run on the proportions of total respondents who like a certain colour? Like below:

red,blue,green,yellow

0.273,0.236,0.182,0.309


Where, of course, 0.273+0.236+0.182+0.309=1.

If the chi square test is not suitable in this case, what test would be? Thanks!

Edit: I tried @Roman Luštrik answer below, and got the following output, why am I not getting a p-value and why does R say "Chi-squared approximation may be incorrect"?

> chisq.test(c(0,0,0,8,6,2,0,0),p = c(0.406197174,0.088746395,0.025193306,0.42041479,0.03192905,0.018328576,0.009190708,0))

Chi-squared test for given probabilities

data:  c(0, 0, 0, 8, 6, 2, 0, 0)
X-squared = NaN, df = 7, p-value = NA

Warning message:
In chisq.test(c(0, 0, 0, 8, 6, 2, 0, 0), p = c(0.406197174, 0.088746395,  :
Chi-squared approximation may be incorrect

-
In the second case, are you assuming you know the total sample size? Or not? – cardinal Apr 16 '11 at 22:04
@cardinal: yes I do know the total sample size. – hpy Apr 17 '11 at 0:30
then just multiply the proportions by the total sample size to transform into a table of counts, and apply the chi-sq. method corresponding to your first example. – Aaron Apr 17 '11 at 0:53
I suspect you are asking about the "goodness of fit" test (using the chi square). The use of which was explained bellow. Cheers, Tal – Tal Galili Apr 18 '11 at 4:42

Correct me if I'm wrong, but I think this can be done in R using this command

> chisq.test(c(15,13,10,17))

Chi-squared test for given probabilities

data:  c(15, 13, 10, 17)
X-squared = 1.9455, df = 3, p-value = 0.5838


This assumes proportions of 1/4 each. You can modify expected values via argument p. For example, you think people may prefer (for whatever reason) one color over the other(s).

> chisq.test(c(15,13,10,17), p = c(0.5, 0.3, 0.1, 0.1))

Chi-squared test for given probabilities

data:  c(15, 13, 10, 17)
X-squared = 34.1515, df = 3, p-value = 1.841e-07

-
I suspect you're seeing this because of some low cell counts (some books I've read suggest a min. of 5 per cell). Maybe someone more knowledgeable on the subject can chip in? – Roman Luštrik Apr 18 '11 at 23:48
Also notice that you can get a p value if you make the last of your probability more than zero (but the warning still remains). – Roman Luštrik Apr 18 '11 at 23:50
Ott & Longnecker (An introduction to statistical methods and data analysis, 5th edition) state, on page 504, that each cell should be at least five, to use the approximation comfortably. – Roman Luštrik Apr 18 '11 at 23:55
@penyuan : You should've mentioned that you have quite some zero counts. Roman is right, using a Chi-square in this case just doesn't work for the reasons he mentioned. – Joris Meys Apr 19 '11 at 0:40
@penyuan : I added an answer giving you some options. – Joris Meys Apr 19 '11 at 22:22

Using the extra information you gave (being that quite some of the values are 0), it's pretty obvious why your solution returns nothing. For one, you have a probability that is 0, so :

• $e_i$ in the solution of Henry is 0 for at least one i
• $np_i$ in the solution of probabilityislogic is 0 for at least one i

Which makes the divisions impossible. Now saying that $p=0$ means that it is impossible to have that outcome. If so, you might as well just erase it from the data (see comment of @cardinal). If you mean highly improbable, a first 'solution' might be to increase that 0 chance with a very small number.

Given :

X <- c(0,0,0,8,6,2,0,0)
p <- c(0.406197174,0.088746395,0.025193306,0.42041479,0.03192905,0.018328576,0.009190708,0)


You could do :

> p2 <- p + 1e-6
> chisq.test(X,p2)

Pearson's Chi-squared test

data:  X and p2
X-squared = 24, df = 21, p-value = 0.2931


But this is not a correct result. In any case, one should avoid using the chi-square test in these borderline cases. A better approach is using a bootstrap approach, calculating an adapted test statistic and comparing the one from the sample with the distribution obtained by the bootstrap.

In R code this could be (step by step) :

# The function to calculate the adapted statistic.
# We add 0.5 to the expected value to avoid dividing by 0
Statistic <- function(o,e){
e <- e+0.5
sum(((o-e)^2)/e)
}

# Set up the bootstraps, based on the multinomial distribution
n <- 10000
bootstraps <- rmultinom(n,size=sum(X),p=p)

# calculate the expected values
expected <- p*sum(X)

# calculate the statistic for the sample and the bootstrap
ChisqSamp <- Statistic(X,expected)
ChisqDist <- apply(bootstraps,2,Statistic,expected)

# calculate the p-value
p.value <- sum(ChisqSamp < sort(ChisqDist))/n
p.value


This gives a p-value of 0, which is much more in line with the difference between observed and expected. Mind you, this method assumes your data is drawn from a multinomial distribution. If this assumption doesn't hold, the p-value doesn't hold either.

-
 You might reconsider your first statement, which I do not believe is correct. If $p_i = 0$ for some $i$ and the observed counts are zero (which they better be), then this just reduces to a submodel. The effect is that the number of degrees of freedom is reduced by one for each $i$ such that $p_i = 0$. For example, consider testing for uniformity of a six-sided die (that is $p_i = 1/6$ for $i \leq 6$). But, suppose we (strangely) decide to record the number of times that the numbers $1,\ldots,10$ show up. Then, the chi-square test is still valid; we just sum over the first six values. – cardinal Apr 20 '11 at 12:24 @cardinal : I just described the data, where the expected value is 0 but the observed doesn't have to be. It's what OP gave us (although on second thought it does indeed sound rather irrealistic). Hence adding a little bit to the p value to make it highly improbable instead of impossible will help, but even then the chi-square is in this case invalid due to the large amount of table cells with counts less than 5 (as demonstrated by the code). I added the consideration in my answer, thx for the pointer. – Joris Meys Apr 20 '11 at 13:10 yes, I'd say if $p_i = 0$, but you observe a count for that cell, then you've got more serious problems on your hands, anyways. :) – cardinal Apr 20 '11 at 13:40

Yes, you can test the null hypothesis: "H0: prop(red)=prop(blue)=prop(green)=prop(yellow)=1/4" using a chi square test that compares the proportions of the survey (0.273, ...) to the expected proportions (1/4, 1/4, 1/4, 1/4)

-
Just to confirm, it will also work with expected proportions that are unequal to each other? – hpy Apr 17 '11 at 0:31
the test won't be meaningful unless you know the full sample size. Proportions of 1.0 / 0.0 / 0.0 / 0.0 mean very different things if they are from a sample of size 1 as opposed a sample of size 100. – Aaron Apr 17 '11 at 0:52
Yes, I DO know the total sample size. – hpy Apr 17 '11 at 14:08

The chi-square test is good as long as the expected counts are large, usually above 10 is fine. below this the $\frac{1}{E(x_{i})}$ part tends to dominate the test. An exact test statistic is given by:

$$\psi=\sum_{i}x_{i}\log\left(\frac{x_{i}}{np_{i}}\right)$$

Where $x_{i}$ is the observed count in category $i$. $i\in \{\text{red, blue, green, yellow}\}$ in your example. $n$ is your sample size, equal to $55$ in your example. $p_i$ is the hypothesis you wish to test - the most obvious is $p_i=p_j$ (all probabilities are equal). You can show that the chi-square statistic:

$$\chi^{2}=\sum_{i}\frac{(x_{i}-np_{i})^{2}}{np_{i}}\approx 2\psi$$

In terms of the observed frequencies $f_{i}=\frac{x_{i}}{n}$ we get:

$$\psi=n\sum_{i}f_{i}\log\left(\frac{f_{i}}{p_{i}}\right)$$ $$\chi^{2}=n\sum_{i}\frac{(f_{i}-p_{i})^{2}}{p_{i}}$$

(Note that $\psi$ is the effectively the KL divergence between the hypothesis and the observed values). You may be able to see intuitively why $\psi$ is better for small $p_{i}$, because it does have a $\frac{1}{p_{i}}$ but it also has a log function which is absent from the chi-square, this "reigns in" the extreme values caused by small expected counts. Now the "exactness" of this $\psi$ statistic is not as an exact chi-square distribution - it is exact in a probability sense. The exactness comes about in the following manner, from Jaynes 2003 probability theory: the logic of science.

If you have two hypothesis $H_{1}$ and $H_{2}$ (i.e. two sets of $p_i$ values) that you wish to test, each with test statistics $\psi_{1}$ and $\psi_{2}$ respectively, then $\exp\left(\psi_{1}-\psi_{2}\right)$ gives you the likelihood ratio for $H_{2}$ over $H_{1}$. $\exp\left(\frac{1}{2}\chi_{1}^{2}-\frac{1}{2}\chi_{2}^{2}\right)$ gives an approximation to this likelihood ratio.

Now if you choose $H_{2}$ to be the "sure thing" or "perfect fit" hypothesis, then we will have $\psi_{2}=\chi^{2}_{2}=0$, and thus the chi-square and psi statistic both tell you "how far" from the perfect fit any single hypothesis is, from one which fit the observed data exactly.

Final recommendation: Use $\chi_{2}^{2}$ statistic when the expected counts are large, mainly because most statistical packages will easily report this value. If some expected counts are small, say about $np_{i}<10$, then use $\psi$, because the chi-square is a bad approximation in this case, these small cells will dominate the chi-square statistic.

-
I'm pretty sure the expected frequencies can't be larger than 10. :) – cardinal Apr 17 '11 at 0:59
@cardinal - glad that this was your objection - for it means the rest of my answer must have been good :). – probabilityislogic Apr 17 '11 at 1:04
Wow, I hope I'm not getting a reputation for being so picky/grumpy. – cardinal Apr 17 '11 at 1:46
I'm don't quite follow the "exact" terminology. Perhaps that is particular to Jaynes' work. Your $\psi$ is the log-likelihood-ratio test statistic though and so $2 \psi$ is asymptotically distributed as a $\chi^2$ distribution by Wilks' theorem. Also, $\chi^2 - 2 \psi \to 0$ in probability, which by Slutsky's theorem is enough to conclude that $\chi^2$ has the same distribution as $2\psi$. Finally, it turns out that $\chi^2$ is the scorte test statistic in this problem as well, which provides another connection between the two test statistics. – cardinal Apr 17 '11 at 1:53
Also, Agresti (Categorical Data Analysis, 2nd ed., p. 80) claims that $\chi^2$ actually converges to a chi-squared distribution faster than $2 \psi$, which seems at odds with your recommendation. :) – cardinal Apr 17 '11 at 1:55
$$\sum_{i=1}^{n} \frac{(O_i - E_i)^2}{E_i}$$
If you write $o_i = \dfrac{O_i}{n}$ and $e_i = \dfrac{E_i}{n}$ to have proportions, where $n=\sum_{i=1}^{n} O_i$ is the sample size and $\sum_{i=1}^{n} e_i =1$, then the test statistic is is equal to
$$n \sum_{i=1}^{n} \frac{(o_i - e_i)^2}{e_i}$$