0
$\begingroup$

I was wondering which test I should use for comparing two proportions from the same survey.

Example: A survey of 1000 people ask two question. "Do you smoke everyday?" and "Do you drink alcohol everyday?" 25% report they smoke cigarettes. 20% report they drink alcohol. Are the two proportions different?

If anyone could provide me with the correct test and a reference I would much appreciate it.

$\endgroup$
0

1 Answer 1

0
$\begingroup$

If you can produce a 2x2 table then you should use McNemar's test. If not then you would have to make the assumption that alcohol and cigarette consumption are independent to do any other analysis.

Further explanation

In response to comment from @SalMangiafico

https://en.wikipedia.org/wiki/McNemar's_test describes McNemar's test as determining whether row- and column- marginal frequencies are equal in a 2x2 table. So in this case you tabulate according to smoking and drinking.

╔══════════════╦═══════╦══════════════╦═══════╗
║              ║ Smoke ║ Do not smoke ║ Total ║
╠══════════════╬═══════╬══════════════╬═══════╣
║ Drink        ║ a     ║ b            ║   200 ║
║ Do not drink ║ c     ║ d            ║   800 ║
╠══════════════╬═══════╬══════════════╬═══════╣
║ Total        ║ 250   ║ 750          ║  1000 ║
╚══════════════╩═══════╩══════════════╩═══════╝

From the information you have provided it is not possible to populate this table, but it may be available to you. But let's suppose that 100 drink but do not smoke and 150 smoke but do not drink:

╔══════════════╦═══════╦══════════════╦═══════╗
║              ║ Smoke ║ Do not smoke ║ Total ║
╠══════════════╬═══════╬══════════════╬═══════╣
║ Drink        ║ 100   ║ 100          ║   200 ║
║ Do not drink ║ 150   ║ 650          ║   800 ║
╠══════════════╬═══════╬══════════════╬═══════╣
║ Total        ║ 250   ║ 750          ║  1000 ║
╚══════════════╩═══════╩══════════════╩═══════╝

We can use mcnemar.test in R:

> mcnemar.test(matrix(c(100, 150, 100, 650), ncol = 2))

    McNemar's Chi-squared test

data:  matrix(c(100, 150, 100, 650), ncol = 2)
McNemar's chi-squared = 9.604, df = 1, p-value = 0.001942

In this case it finds there is a statistically significant difference between the proportions smoking and the proportions drinking.

In fact, across all valid values of a (from 0 to 200) you find that there is a statistically significant difference.

If you do not have access to the 2x2 table, then a publication from 1993 suggests an approximate solution C. J. Wild and G. A. F. Seber. Comparing Two Proportions from the Same Survey. The American Statistician Vol. 47, No. 3 (Aug., 1993), pp. 178-181

$$ \left|z\right| \ge \frac{\hat{p}_1 - \hat{p}_2}{\sqrt{\max(\hat{p}_1+\hat{p}_2, \hat{q}_1+\hat{q}_2)/n}} $$

So in your case:

$$ \left|z\right| \ge \frac{0.25 - 0.2}{\sqrt{\max(0.25+0.2, 0.75+0.8)/1000}} = \frac{0.05}{\sqrt{1.55/1000}} = 1.27 $$

This does not exceed the critical value of 1.96, so you conclude there is insufficient evidence to conclude the proportions are different. Comparing this to the finding above suggests that this method may be overly conservative.

$\endgroup$
5
  • 2
    $\begingroup$ McNemar's test doesn't address the question about the two proportions. $\endgroup$ Commented Jan 4, 2020 at 14:17
  • $\begingroup$ 1. Thanks for expanding your answer, but, honestly, McNemar's test still doesn't address the question asked (at least in the form it is written). First, note that you've added information that isn't included in the question: creating counts for "Drink & Smoke", "Drink & Don't smoke", and so on. But more importantly, note that McNemar's test in the table you've designed doesn't compare the 250 count to the 200 count. It compares the 150 count to the 100 count (and ignore the other counts!). This wouldn't address the question which asked about the 25% and 20%; It answers a different question. $\endgroup$ Commented Jan 6, 2020 at 11:41
  • $\begingroup$ 2. Also, please note that the results from R from your example and the results from the hand calculation don't match. I'm not familiar with the 1993 paper cited, but the fact that the results are very different certainly don't suggest confidence that the two are addressing the same situation. $\endgroup$ Commented Jan 6, 2020 at 11:44
  • $\begingroup$ @SalMangiafico yes I appreciate that I introduced information that was not available, but then I also stated that you can perform a sensitivity analysis by varying over all possible values for the interaction between smoking and drinking and you still find statistical significance (although the p-values do vary). Your complaint about the McNemar test only using the off diagonal entries is a common criticism but I still think this is the most appropriate test. The 1993 paper uses a lower bound approximation and will be conservative. $\endgroup$
    – tristan
    Commented Jan 6, 2020 at 22:18
  • $\begingroup$ I think we can use 25% and 20% in Mcnemar's test as long as they are proportions of the whole sample size. $\endgroup$
    – user314428
    Commented Mar 15, 2021 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.