14
$\begingroup$

I have one sample with n=170 and two binary variables (A,B) that can take as a value 1 or 0, where 1 counts as a success and 0 counts as a failure. What I want to know is whether the means of these two variables are equal.

To find this out I generate a new variable that takes the difference between these two variables called C, so C = B-A. I then compute the p-value for the hypothesis that C is normally distributed with the Shapiro-Wilk test and I find a p-value of .96, so I choose not to reject this hypothesis. Apart from that the difference is normally distributed, I am not worried about the other assumptions required for a paired t-test.

Question: Can I use the paired t-test in this circumstance or is it a mistake to use the Shapiro-Wilk test for binary data to check for normality and should I use the Wilcoxon sign rank test instead?

I would much prefer to use the t-test, because I believe it has a higher power than the Wilcoxon sign rank test, but that higher power pretty much does not matter if the test used is the wrong one.

Cheers,

Martin

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ Something is the matter with your application of the S-W test: because the values of $C$ must be one of the three elements $\{-1,0,1\},$ this test is certain to find that the distribution is non-normal. The test, however, is irrelevant, because the underlying distribution of $C$ does not matter; only the sampling distribution of the mean of $C$ matters, and that will be very close to Normal except in extreme cases. $\endgroup$
    – whuber ♦
    Commented Oct 23, 2013 at 14:14
  • $\begingroup$ Yes that is why I am asking. I used the command swilk in stata to test whether C comes from a normal distribution, but I also know how that looks when plotted against the normal distribution. So that is why I am a bit hesistant of using the t-test. As I understand you though, I can use the t-test because the sampling distribution of the mean of C is probably normal for this sample size? $\endgroup$
    – Martin
    Commented Oct 23, 2013 at 14:22

3 Answers 3

Reset to default
13
$\begingroup$

If I understand the context correctly, then McNemar's test is exactly what you want. It compares two binomial variables measured in each subject, sort of a paired chi-square test. The key point is that your data are paired -- you've measured two different binomial outcomes in each subject, so need a test that accounts for that.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks, but why can you not use a paired t-test or z-test of proportions as the other two answers suggested below? Perhaps that I should be more specific about the context. I have one case file and I have three experts look at it. Experts 1,2,3. What I want to know is whether the first expert agrees as often with the second expert as she does with the third expert. Whether she agrees with the second expert is independent from whether or not she agrees with the third expert. $\endgroup$
    – Martin
    Commented Oct 23, 2013 at 18:58
  • 1
    $\begingroup$ This new, completely different, question cannot be tackled by any of the tests discussed here. Agreement is not implied by marginal homogeneity. $\endgroup$
    – Michael M
    Commented Oct 23, 2013 at 19:46
  • $\begingroup$ Thank you, but I am not testing for agreement. I am trying to find out whether Adam and Claire agree more often with one another than Bertha and Adam on whether a painting is worth buying and I have a sample of their answers for 170 paintings. Let's say that if Adam and Bertha agree they buy a reproduction of the painting and that if Adam and Claire agree they buy a reproduction of the painting. Now if I am selling reproductions of paintings, would I rather have that Adam and Bertha come into my store or that Adam and Claire come into my store or am I indifferent? That's what I want to know. $\endgroup$
    – Martin
    Commented Oct 23, 2013 at 20:01
  • $\begingroup$ Quick intro to chi square vs McNemar test: theanalysisfactor.com/… $\endgroup$
    – theforestecologist
    Commented May 2, 2021 at 19:51
4
$\begingroup$

There's a few things here.

  1. For binomial data, the variance is directly determined by the mean, and isn't an additional parameter, so there's no need to do a t-test... a normal z-test is slightly more efficient.
  2. For binomial data, the Normal approximation (i.e. a Wald test) often fails. See Agresti and Coull, 1998, for some more detailed discussion and simulation studies. http://www.stat.ufl.edu/~aa/articles/agresti_coull_1998.pdf

They give some recommendations about when it's okay to use or not use the normality assumption (as do others)... generally the closer you get to p=.5, and the larger your data set, the better it is, the further away from .5 you get (towards p=0 or p=1), or the smaller the data, it's worse.

But the Wilcoxon sign rank test is popular for this kind of data.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Though I will have to look at how close .14 is to p=0. $\endgroup$
    – Martin
    Commented Oct 23, 2013 at 15:12
  • 3
    $\begingroup$ The standard test in such a setting is McNemar's test. (It is equivalent to an exact version of Wilcoxon's signed test) $\endgroup$
    – Michael M
    Commented Oct 23, 2013 at 15:22
0
$\begingroup$

You are using the term 'mean' but actually you are comparing 'proportions' as your variables are categorical. I would ignore any issues with normality as the sampling distribution of the proportions will be normal (ignoring some pathological situations such as low sample size which is not an issue here or proportions close to $0$ or $100$).

I recommend looking at the two-proportion z-test at the wiki: Common Statistical Tests. Search for "two-proportion z-test" in the table for the relevant test and conditions under which it is valid.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks! Though what do you mean by close to? The proportions are actually between 15 and 40 out of a 100. $\endgroup$
    – Martin
    Commented Oct 23, 2013 at 15:10
  • 1
    $\begingroup$ Your suggestion ignores the pairing in the data. McNemar's test is the appropriate test. $\endgroup$
    – Alexis
    Commented May 4, 2014 at 14:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.