0
$\begingroup$

I have a data that consists of some "after treatment" and "before treatment" observations from some individuals.

Specifically the data looks like this:

        A   B   C    D
before  10  15  4    0
after   12  12  2    3

A = not agree
B = slightly not agree
C = agree
D = definitely agree

Basically I have a group of individuals ($N=29$) who did a survey which consists of answering some questions before and after some intervention. My question is how do I test if there is difference in the "before" and "after" proportion of answering the question.

For example, for the "before" row, there are 10/29 people answers "not agree", but after the intervention, 12/29 people answers "not agree".

I was thinking of the chi-square test, but it is definitely wrong to use chi-square since the samples of "before" and "after" are coming from the same group of individuals. McNemar test won't work since I have more than 2 categories.

Could someone advise as to what test I can use?

|cite|improve this question|||||
$\endgroup$
1
$\begingroup$

The test you are looking for is called a test of marginal homogeneity (the table you give are the two margins of a 4x4 cross-classification, and you want to test if these are equal, hence the name). There is at least one R-packages for the Stuart-Maxwell test, which is a Wald test. A bit more accurate for small samples may be a likelihood ratio test using maximum likelihood estimators, and there are packages by Joe Lang, Antonio Forcina et al., or Wicher Bergsma (package cmm), however these can do much more and just marginal homogeneity so you for a quick test maybe search for the Stuart-Maxwell one.

|cite|improve this answer|||||
$\endgroup$
0
$\begingroup$

I can think of four paired-tests that you could use, for better or worse, in this scenario:

  1. Force the data to work with McNemar's test by combining categories A+B and C+D. This is far from ideal, as you will lose quite a bit of power.
  2. Convert A,B,C,D to 1,2,3,4 and use the sign test. If there are many unsigned pairs (ie, for many respondents the level of agreement does not change after treatment), this is not a good choice.
  3. Convert A,B,C,D to 1,2,3,4 and use the Wilcoxon signed-rank test. This will be useful if you are interested in the direction of change of agreement before and after, rather than if you are interested in just whether or not a change has occurred.
  4. Use the McNemar-Bowker test (no Wikipedia page!) which is an extension of McNemar's test for 3 or more response categories. This only tells you if there is a difference after treatment, not if this difference tends to be in a specific direction (unlike previous options). It ignores the ordinality of your groups.

Check the assumptions for each test are (approximately) met before using them. And, ideally, choose a test a-priori! Don't try all four then pick the one which suits you best.

|cite|improve this answer|||||
$\endgroup$
  • $\begingroup$ Hi wjchulme, it seems it is most appropriate to use the 4th suggestion in your list. But as a side question, if I give the survey to two different groups of people, each consists of N=26 or same number of individuals, and if I want to see if they will answer the questions differently. Does it mean I can assume that each group's distribution comes from a Multinomial distribution and test if the parameter is different? i.e. assuming $X_{1i}$ ~ Multi( 26, $\theta_1$), $X_{2i}$ ~ Multi( 26, $\theta_2$) and test if $\theta_1$ is different from $\theta_2$? $\endgroup$ – john_w Mar 23 '17 at 3:21
  • $\begingroup$ Oh I think in this case $\theta_1$ and $\theta_2$ should be a vector, because Multinomial distribution has more than 1 parameter. So it would be testing if $\theta_1$ (a vector of parameter) is different from $\theta_2$ (a vector of parameter), is it correct? $\endgroup$ – john_w Mar 23 '17 at 3:23
  • $\begingroup$ @john_w you could use a chi-squared test for this if the two different groups of people are independent and unpaired. $\endgroup$ – wjchulme Mar 23 '17 at 11:30
  • $\begingroup$ But do you think it is correct to say "I am comparing the parameter of two multinomial distribution" in that case? Can I say that? and I can use chi-squared test to test "if there is difference of parameter values between the two multinomial distribution". Can I say that? $\endgroup$ – john_w Mar 23 '17 at 11:46
  • $\begingroup$ I just want to confirm if it is conceptually correct to assume the first distribution is multinomial and the second distribution is also multinomial with same sample size. And test to see if the parameter value is different by using a chi-squared test. Is it correct to say those "sentences" conceptually? $\endgroup$ – john_w Mar 23 '17 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.