I have data from an experiment where students were tested on multiple words for the correct pronunciation of a specific linguistic phenomena.
The experiment was done with a control and experimental group, and all students were tested prior to a specific instruction and at the end of the experiments (roughly 8 weeks later).
> head(act1) studentid groupid itemid test type result 1 1B B 1 0 1 0 2 5B B 1 0 1 1 3 6B B 1 0 1 1 4 8B B 1 0 1 0 5 11B B 1 0 1 1 6 15B B 1 0 1 1 > levels(act1$groupid)  "B" "D"
D is the experimental group while B is the control group.
First, I want to test the hypothesis that the experimental group has had a significant improvement in the production of the studied linguistic phenomena.
I ran McNemar's test on each group individually:
> act1wide <- reshape(act1, idvar=c("studentid","groupid","itemid","type"), timevar=c("test"), v.names=c("result"), direction="wide") > act1wideb <- subset(act1wide, groupid=='B') > act1wided <- subset(act1wide, groupid=='D') > mcnemar.test(act1wideb$result.0, act1wideb$result.1) McNemar's Chi-squared test with continuity correction data: act1wideb$result.0 and act1wideb$result.1 McNemar's chi-squared = 0.0556, df = 1, p-value = 0.8137 > mcnemar.test(act1wided$result.0, act1wided$result.1) McNemar's Chi-squared test with continuity correction data: act1wided$result.0 and act1wided$result.1 McNemar's chi-squared = 9.0312, df = 1, p-value = 0.002654
This seems to show that group 'B' did not improve while group 'D' did.
- Is this a valid test?
- How can I have a more meaningful test that would compare group data at the same time?
- The factor type has three levels. How can I break down the analysis to get specific information by type?
I am a bit of a stats noob so I greatly appreciate detailed explanations on how to make sense of my data.