52
$\begingroup$

Suppose that I have three populations with four, mutually exclusive characteristics. I take random samples from each population and construct a crosstab or frequency table for the characteristics that I am measuring. Am I correct in saying that:

  1. If I wanted to test whether there is any relationship between the populations and the characteristics (e.g. whether one population has a higher frequency of one of the characteristics), I should run a chi-squared test and see whether the result is significant.

  2. If the chi-squared test is significant, it only shows me that there is some relationship between the populations and characteristics, but not how they are related.

  3. Furthermore, not all of the characteristics need to be related to the population. For example, if the different populations have significantly different distributions of characteristics A and B, but not of C and D, then the chi-squared test may still come back as significant.

  4. If I wanted to measure whether or not a specific characteristic is affected by the population, then I can run a test for equal proportions (I have seen this called a z-test, or as prop.test() in R) on just that characteristic.

In other words, is it appropriate to use the prop.test() to more accurately determine the nature of a relationship between two sets of categories when the chi-squared test says that there is a significant relationship?

$\endgroup$
23
$\begingroup$

Very short answer:

The chi-Squared test (chisq.test() in R) compares the observed frequencies in each category of a contingency table with the expected frequencies (computed as the product of the marginal frequencies). It is used to determine whether the deviations between the observed and the expected counts are too large to be attributed to chance. Departure from independence is easily checked by inspecting residuals (try ?mosaicplot or ?assocplot, but also look at the vcd package). Use fisher.test() for an exact test (relying on the hypergeometric distribution).

The prop.test() function in R allows to test whether proportions are comparable between groups or does not differ from theoretical probabilities. It is referred to as a $z$-test because the test statistic looks like this:

$$ z=\frac{(f_1-f_2)}{\sqrt{\hat p \left(1-\hat p \right) \left(\frac{1}{n_1}+\frac{1}{n_2}\right)}} $$

where $\hat p=(p_1+p_2)/(n_1+n_2)$, and the indices $(1,2)$ refer to the first and second line of your table. In a two-way contingency table where $H_0:\; p_1=p_2$, this should yield comparable results to the ordinary $\chi^2$ test:

> tab <- matrix(c(100, 80, 20, 10), ncol = 2)
> chisq.test(tab)

    Pearson's Chi-squared test with Yates' continuity correction

data:  tab 
X-squared = 0.8823, df = 1, p-value = 0.3476

> prop.test(tab)

    2-sample test for equality of proportions with continuity correction

data:  tab 
X-squared = 0.8823, df = 1, p-value = 0.3476
alternative hypothesis: two.sided 
95 percent confidence interval:
 -0.15834617  0.04723506 
sample estimates:
   prop 1    prop 2 
0.8333333 0.8888889 

For analysis of discrete data with R, I highly recommend R (and S-PLUS) Manual to Accompany Agresti’s Categorical Data Analysis (2002), from Laura Thompson.

$\endgroup$
  • 2
    $\begingroup$ Is there a common name for the test that prop.test() performs? $\endgroup$ – Atticus29 Apr 26 '13 at 17:00
  • 2
    $\begingroup$ "It is referred to as a z-test". $\endgroup$ – russellpierce Sep 9 '14 at 20:49
  • $\begingroup$ @chl I'm a bit confused - I thought prop.test and chisq.test both use the chi-square, which would explain the identical p values, as well as why in this post on R-Bloggers they have their own ad hoc function. $\endgroup$ – Antoni Parellada Aug 20 '15 at 1:45
  • $\begingroup$ @Antoni Yes, this is what Keith explained in his reply. $\endgroup$ – chl Aug 20 '15 at 9:05
  • 3
    $\begingroup$ whats $n_1$ and $n_2$, $f_1$, $f_2$, $p_1$, $p_2$? $\endgroup$ – tomka Sep 6 '16 at 19:28
22
$\begingroup$

A chi-square test for equality of two proportions is exactly the same thing as a $z$-test. The chi-squared distribution with one degree of freedom is just that of a normal deviate, squared. You're basically just repeating the chi-squared test on a subset of the contingency table. (This is why @chl gets the exact same $p$-value with both tests.)

The problem of doing the chi-squared test globally first and then diving down to do more tests on subsets is you won't necessarily preserve your alpha -- that is, you won't control false positives to be less than 5% (or whatever $\alpha$) across the whole experiment.

I think if you want to do this properly in the classical paradigm, you need to identify your hypotheses at the outset (which proportions to compare), collect the data, and then test the hypotheses such that the total threshold for significance of each test sums to $\alpha$. Unless you can prove a priori that there's some correlation.

The most powerful test for equality of proportions is called Barnard's test for superiority.

$\endgroup$
  • $\begingroup$ @gung I'm a bit confused - I thought prop.test and chisq.test both use the chi-square, which would explain the identical p values, as well as why in this post on R-Bloggers they have their own ad hoc function. $\endgroup$ – Antoni Parellada Aug 20 '15 at 1:50
  • $\begingroup$ I don't understand what's confusing you, @AntoniParellada. This answer states that they are "exactly the same", which makes sense if the "both use the chi-square". $\endgroup$ – gung Aug 20 '15 at 1:58
  • $\begingroup$ @gung in the original answer chl states that prop.test() ... is referred to as a z-test in contradistinction to chisq.test(). Later Keith says, "A chi-square test for equality of two proportions is exactly the same thing as a z-test. (This is why @chl gets the exact same p-value with both tests.)" $\endgroup$ – Antoni Parellada Aug 20 '15 at 2:02
  • 1
    $\begingroup$ That seems to be just an infelicitous phrasing, @AntoniParellada. Conceptually, the 2 tests are distinct, which is what I discussed in my other answer you saw. But mathematically, they are equivalent. In fact, I believe the R function prop.test() actually just calls chisq.test() & prints the output differently. $\endgroup$ – gung Aug 20 '15 at 2:07
  • $\begingroup$ @gung I had been working on a function similar to R-Bloggers and I was going to make a post for people at my beginner's level, quoting you in fact regarding a few key concepts that your have written, running through the math both for chi square and z-test, and then giving the R code. $\endgroup$ – Antoni Parellada Aug 20 '15 at 2:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.