I have a question on selecting the appropriate statistical test as below:
I have a group of people attending a training course, in order to measure whether the training course is effective in improving attendees' knowledge. We did the following:
- attendees will complete a pre-test (TRUE or FALSE only) before the training
- attendees attending the training
- attendees will complete a post-test (TRUE or FALSE only) after training
Both pre and post tests are having exactly the same questions, the test is tagged with barcode so that we are able to trace which pre and post tests are a pair.
By further breaking down the research question, it can be divided into two parts:
1. whether the training is effective in general
2. whether the training can improve answering of a particular question (item)
For question 1, I think paired t-test will be appropriate: we mark all the test, each attendee will have a pre-test score and a post-test score, we take mean of the pre-test and post-test score respectively, and then go for paired t-test.
For question 2, at first I am thinking of doing a McNemar’s test: as the answer for each question in pre-test and post-test can be matched. I planned to do something like this. But then I have received some alternative method which I'm not sure whether it is appropriate or not:
- count the number of attendees answered question X correctly in pre-test and divide it with total number of attendees answer question X, so it becomes a percentage
- do the same thing in post-test, so we have two percentages here
- perform "paired" t-test on it
When I first saw it, I wonder how I can "pair" each question, as the categorical data is already converted into continuous data. Perhaps the proposer accidentally add the word "pair" and he actually means "t-test" only. But I still wonder why discard the information based on pairing.
A few questions here:
- for question 1, is using paired t-test appropriate?
- for question 2, is McNemar's test appropriate?
- for question 2, is "t-test" still appropriate? is there any reason why "t-test" is prefered?