I have been pondering over this question for a couple of days now and maybe got lost in the details. I am a Psych Grad student, so my stats knowledge is not as deep as I would wish for.
I did an experiment with the following design: participants watched some videos and answered questions about them afterwards. All participants answered the same questions. One of the questions, the dependent variable, is dichotomous. Did they answer the question correctly or not (1/0). The other question of interest (IV1) is dichotomous too and of the same kind, just asking for another aspect of the videos. And the last Variable is about the category of video the question is targeting (1/2/3/4). So:
- DV: dichotomous
- IV1: dichotomous
- IV2: categorical
My hypothesis is that people performing well in IV1 in category 4 of IV2 will perform worse for the DV.
As the counts of DV anf IV1 depend on the performance of the participants, the design is unbalanced.
In every single paper of my field and specific research subject, everyone uses repeated measures ANOVAS. And reading a little, it looks like the thing I could do is an unbalanced Type-II-SS ANOVA. On the other hand I found a paper (here) that says ANOVA for dichot. variables is okay, but the design needs to be balanced, if I got it right. And in general, an ANOVA will only show me THAT there is an impact of the IV on the DV, not in which direction. So - when doing this in R - could I first create the linear model
mod1 <- lm(DV ~ IV1, IV2)
and then put this into Anova(mod1, type = 2) and see what kind of effects I find? And if I find one, lets say for IV1, then I can suggest the direction based on the intercept of the lm?
For the post-hoc tests, I would do pairwise t-tests and I read somewhere that in R glth() of the multicomp-package could do the job.
I also thought of correlations using the phi-coeff. and chi². But then again, I'd have several correlations and would need to compare those too somehow.
So: Is an unbalanced ANOVA the way to go for me? All papers before me did it, I could just do it. But I want to make sure it really is the correct approach, statistically. Another suggestion I got was doing a logistic regression. But as far as I read about it, this won't work with my kind of IVs. And if I do the ANOVA, can I do the lm approach to tell something about the direction of effect?