6
$\begingroup$

I read in several books as well as in different posts (e.g. here) that independence is required for two-sample proportions z tests. But so far I could not find an explanation why this is the case and what would happen if such a test were used for comparing proportions in non-independent data.

As an example: All participants answered two questions. One question was answered correctly by 85% and the other question was answered correctly by 65%. I am interested in whether the proportion of correct answers is significantly larger for the first than the second question.

Therefore, I would like to understand:

  1. Why is it wrong to use a two-proportions z test in this case? (Does it also depend on the question one would like to answer with the statistical test?)

  2. What are the consequences of using the procedure nonetheless (e.g. will the significance values be systematically too high or low)?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

All participants answered two questions. One question was answered correctly by 85% and the other question was answered correctly by 65%. I am interested in whether the proportion of correct answers is significantly larger for the first than the second question.

That would be a paired test.

Why is wrong to use a two-proportions z test in this case?

Because the independent-sample proportions test relies on ... independence. Specifically, the (normal approximation of the) distribution of the test statistic under the null hypothesis is computed on the basis that the observations are independent.

Does it also depend on the question one would like to answer with the statistical test?

No, at least not for any of the questions that occur to me.

What are the consequences of using the procedure nonetheless (e.g. will the significance values be systematically too high or low)?

If you do it with samples that are paired (and so positively correlated within the pairs), as in your example, then the variance of the difference in proportions will be different from what the independence assumption would suggest.

As a result, your true significance level will be larger than you chose it to be so you'll reject more often (much more often) than you should.

Below is the results of a simulation, first when the two columns are independent, and second when the variables are correlated (to get correlated binary variables I generated correlated standard normals with $\rho=0.6$ and dichotomized them by recording $1$ if they were less than 0.1**; the independent variables were created the same way but from independent normals).

** I chose a $p$ that was not exactly 1/2, in case there was any thought that $p$=1/2 might be a special case

enter image description here

These are 10000 simulations at n=100 for a two-tailed two sample proportions test (here done via a chi-square using R's default settings; the chi-square should be the square of the z-test done with the same settings). The true distribution of the test statistic is discrete and the chi-square (and the corresponding z-test) is approximate. The small spike in the left-side plot is due to that discreteness (and leads to mild conservatism in the test with independent proportions); ideally it should look uniform. In the right hand plot, correlated binaries (as described above) were used. There, about 98% of the tables generated had p-value <0.05. This is when the null hypothesis is true.

A small amount of effect might be tolerable, but this is quite dramatic.

Improve this answer
$\endgroup$
10
  • $\begingroup$ Ok, so this would mean to use something like McNemar test I suppose. Do I understand you right, using a test that assumes independence would yield a more conservative result with respect to whether or not H1 is accepted? $\endgroup$
    – grey
    Commented Dec 29, 2014 at 14:37
  • 2
    $\begingroup$ McNemar's test would be the usual way I think, though one might set it up as a GLM. Yes, more conservative when there's positive dependence. $\endgroup$
    – Glen_b
    Commented Dec 29, 2014 at 15:25
  • $\begingroup$ With respect to GLM, do you think of a generalized linear mixed model? I also thought about that, especially as I have a second, independent variable in my design. But I feel like cracking a nut with a sledgehammer to use such a procedure for my (in my opinion rather simple) questions (and I imagine that the required sample sizes are rather large...). Is it really adequate to think about such rather complex procedures with simple proportions? $\endgroup$
    – grey
    Commented Dec 29, 2014 at 19:28
  • 1
    $\begingroup$ Yes, you'd probably want to use a GLMM in that situation. When you're dealing with paired 0/1 data with a covariate, it's not all that simple. If the additional covariate is dichotomous (or similarly simple in form) you might be able to do something else (construct an extension of a McNemar test perhaps), but it's not necessarily much simpler. $\endgroup$
    – Glen_b
    Commented Dec 29, 2014 at 21:12
  • $\begingroup$ The second variable is indeed dichotomous. Hence, I have one within (the 2 questions) and one between (additional information y/n) subjects manipulation. I searched the internet and statistics books for something like a 2x(2x2) contingency table that accounts for the partial dependency, but I haven't found a solution so far. Perhaps your idea with the extension of the McNemar test comes close to that? I also thought of calculating a chi square for two 2x2 tables (1: question 1 corr/incorr, add. info y/n; 2: the same for question 2) and then compare whether the chi square values are different. $\endgroup$
    – grey
    Commented Dec 29, 2014 at 21:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.