# Controlled experiment with binary outcome variable

I have data from an experiment where students were tested on multiple words for the correct pronunciation of a specific linguistic phenomena.

The experiment was done with a control and experimental group, and all students were tested prior to a specific instruction and at the end of the experiments (roughly 8 weeks later).

> head(act1)
studentid groupid itemid test type result
1        1B       B      1    0    1      0
2        5B       B      1    0    1      1
3        6B       B      1    0    1      1
4        8B       B      1    0    1      0
5       11B       B      1    0    1      1
6       15B       B      1    0    1      1

> levels(act1$groupid) [1] "B" "D"  D is the experimental group while B is the control group. First, I want to test the hypothesis that the experimental group has had a significant improvement in the production of the studied linguistic phenomena. I ran McNemar's test on each group individually: > act1wide <- reshape(act1, idvar=c("studentid","groupid","itemid","type"), timevar=c("test"), v.names=c("result"), direction="wide") > act1wideb <- subset(act1wide, groupid=='B') > act1wided <- subset(act1wide, groupid=='D') > mcnemar.test(act1wideb$result.0, act1wideb$result.1) McNemar's Chi-squared test with continuity correction data: act1wideb$result.0 and act1wideb$result.1 McNemar's chi-squared = 0.0556, df = 1, p-value = 0.8137 > mcnemar.test(act1wided$result.0, act1wided$result.1) McNemar's Chi-squared test with continuity correction data: act1wided$result.0 and act1wided\$result.1
McNemar's chi-squared = 9.0312, df = 1, p-value = 0.002654


This seems to show that group 'B' did not improve while group 'D' did.

Questions:

• Is this a valid test?
• How can I have a more meaningful test that would compare group data at the same time?
• The factor type has three levels. How can I break down the analysis to get specific information by type?

I am a bit of a stats noob so I greatly appreciate detailed explanations on how to make sense of my data.

-