1
$\begingroup$

I am trying to determine the most appropriate test for parts of my dataset. I have been using a $\chi^2$ test, but I am now realising it might not be the most appropriate. I am using Stata.

I have a survey consisting of five-point Likert scales, i.e. Definitely not, Probably not, Unsure, Probably, Definitely. I have 233 respondents answering many different questions.

As an example, these questions ask:

  • Do you think X should be available?
  • Do you think Y should be available?
  • Do you think Z should be available?

I would like to know if there is a significant difference in respondents' beliefs in the availability of X and the availability of Y. Here is an example crosstab:

enter image description here

(N.B. most of the responses to the questions are much less unanimous than this).

When I do a $\chi^2$ test in Stata (I did not use Fishers due to sample size), it says there is a significant difference with a p-value of 0.00000000000000001114 (!!!). This seems rather unlikely to me (just look at the values!). I tried Stuart-Maxwell using the symmetry command and got a much more reasonable p-value of 0.0007. I used Stuart-Maxwell instead of McNemar as it is not 2x2.

Should I use $\chi^2$ (as I have been doing)? I feel like I shouldn't be. Also while the data is not unpaired it is not repeated measures, it is asking respondents about their attitudes to two different things and I would like to know whether there is a significant difference.

Which is the most appropriate test to use? One of these or some other test I haven't considered?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

The chi-squared test is not going to help you here. In technical terms, chi-squared is a test of independence. It's checking to see if there is a relationship between views about X and views about Y, not what that relationship is (same vs. different). Also, many of your cells are empty, which is a red flag signaling that chi-squared is not a good choice.

Depending on your goals, your quickest bet might be to re-code your data. For example, you can measure how many respondents had the same response about X and Y (that is, how many people gave answers on the main diagonal of your table) and how many had different responses about X and Y (upper right and lower left cells). You'd need to make a judgement call about how to code some of the less extreme combinations. You could then use something like a binomial test to determine if the proportion whose views on X and Y agreed differed from the proportion whose views on X and Y disagreed.

Your categorical variables are ordered, so I think that a generalized linear model may give you better information if you want to know more about the relationship between views on X and views on Y. Agresti's text on categorical data analysis is a good place to learn about applying this type of model to categorical data.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The method described using a binomial test is essentially the classic sign test. In this test, the ties are usually excluded from the test, including from being counted as part of the sample size. You could think of this as starting with the trinonomial data (+, 0, -), and throwing away all the 0's so that you can perform a binomial test on the +'s and -'s. This is similar in spirit to McNemar's test, which can also be reduced to a binomial test, and in which the concordant cells don't enter into the analysis at all. $\endgroup$
    – Sal Mangiafico
    Commented Jul 31, 2018 at 18:06
  • $\begingroup$ So, if I got this straight, for the table presented, using the sign test approach, 213 are on the main diagonal, so we ignore those, and we have X < Y = 3 and X > Y = 17. So we can use binom.test(3, 20) or binom.test (17, 20) for the results p = 0.0026; odds ratio = 17 / 3 = 5.7. $\endgroup$
    – Sal Mangiafico
    Commented Jul 31, 2018 at 18:14
0
$\begingroup$

Your stated question is: "it is asking respondents about their attitudes to two different things and I would like to know whether there is a significant difference."

First, you should probably be treating your data as ordinal in nature, so chi-square or mcnemar-style tests won't be the most appropriate.

Also, you want a test that is taking into account the paired nature of the responses (as the same person is answering the question about X and about Y).

Depending on the capabilities of the software you are using, ordinal regression would be an ideal approach, such as, Likert ~ Question + (1|Subject), where Likert is the response (DV), Question is a factor with values X or Y (IV), and Subject is the id of the individual (random blocking effect). An advantage of this approach is that you could put all the questions (X, Y, Z) into one analysis, and, again depending on your software, make multiple comparisons among the questions.

A simpler nonparametric approach would be the Wilcoxon signed-rank test, essentially taking the difference in Likert scores between X and Y for each individual and comparing these differences to zero. This test, however, assumes that interval values (i.e. numeric numbers) can be assigned to the response categories.

Finally, a (two-sample, paired) sign test could be used, with fewer assumptions than the Wilcoxon test. As commented in their answer, this is essentially the approach outlined by @AndrzejW .

It is a good idea to understand the hypotheses and assumptions of whichever method you are considering moving forward.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.