# Am I using the appropriate statistical comparison tests & applying them correctly to my qualitative coded data?

I am trying to determine if I am using the correct statistical tests in making comparisons & finding out if there are any statistically significant shifts of student attitudes. I have three open response questions from 3 consecutive academic terms (fall, winter, spring).

I used inductive qualitative coding to determine coding categories, which include: 'confident', 'not confident', 'interested/engaged', & 'not engaged'. As the questions were open ended, responses varied in what feeling/theme they discussed. Some would discuss their confidence, some would indicate if they were interested in doing the task, a few discussed both, and a few discussed neither. As such, the codebook categories are not exclusive except for the 'confident'/'not confident' & 'interested/engaged'/'not engaged' pairings

I'm trying to see if there is some significant shift (whether positive or negative) between student attitudes between the terms. My surveyed populations for the fall, winter, & spring are 67, 51, & 43 respectively. I also have matched data pairings between fall & winter (N=34) and for all terms (fall, winter, & spring, N=21)

While hindsight shows me that having more questions that focus on confidence and engagement separately, this was a newer research project & I can't go back to change the past.

Because my data is better described as categorical than discrete (although you can argue to use tests relevant to discrete data), I have been looking into using the following statistical tests:

• Fisher's exact for testing all data between fall & winter and winter & spring
• McNemar's test for testing matched pair data between fall & winter and winter & spring
• Chi-squared test for testing all of the data from all three terms
• Cochrane Q for the matched data across all three terms

This is the reference I've been using to guide me in choosing the type of tests depending on the scenario: https://www.bristol.ac.uk/medical-school/media/rms/red/which_test.html[\ ]

I'm still very much new to these statistical test, and have been reading into them for the past few weeks. But I wanted to give enough context to aid in finding helpful answers.

The things I am uncertain or confused about are:

1. Am I using the correct statistical comparison tests?
2. When creating/using contingency tables, should my axes be comparing the number of 'confident' & 'not confident' instances together? (since they are mutually exclusive, despite not every response referencing confidence in any form). For example, should my table be:
Fall Winter
Confident a b
Not confident c d

or

'Confident' Fall Winter
1 a b
0 c d

(where 1 is a response that indicated they felt confidence & 0 meant no mention of any positive confidence)

I think the tests you have chosen are fine. Some might suggest multinomial logistic regression as an alternative but the tests you have chosen will probably work for your purposes.

Data input for Fisher's exact test (and chi-square test, and McNemar's test) may depend on the software you use, but your first table version is the typical one for chi-square and Fisher's exact test in my experience. I assume the letters would be replaced with counts, such as

             Fall Spring
Confident      10      7
NotConfident   12      5


For McNemar's test, you probably want something like this:

             Fall
Spring         Confident NotConfident
Confident           30           40
NotConfident        12           18


The above should work in R.

In SPSS, I'm used to inputting raw data for chi-square test and Fisher's test, i.e.

id conf  time
1  1     1
2  0     2
3  1     2
...
#where conf 1 = confident, conf 0 = not confident and time 1 = fall and time 2 = spring and id=participant id.


For McNemar's test in SPSS, this is a good guide.