I recently completed a research project that involved motor learning in control (non-disease) subjects. The outcome of the motor learning is binary: 1 - subject learned, 0 - subject didn't learn. 13 out of 30 (43%) of control subjects learned. I ran a best-subset logistic regression procedure and ended up with a good model that predicted the outcome correctly 88% of the time (on a test set). The model had two continuous variables that reflect motor performance, A and B.
Now, I am expanding these findings to a disease group. 2 of 20 (10%) of the disease subjects learned. With a Fisher's test that is a significant difference from 13 out of 30. I want to know if it is the disease or the motor performance that causes the reduced learning in the disease group. The problem is that my disease group is older than my control group. (t(48)=-2.3, p=0.03). Of course, variable A is correlated with age, but variable B is not.
Here's why I've gotten myself into a circular argument:
If I compare the control and disease group on variables A and B with age as a covariate, there is no difference (A; F(1,47) = 1, p = 0.3) (B; F(1,47)= 4, p=0.05).
The original model (Learning ~ A + B) works well if I just include the data from the disease group without adding diagnosis to the model. That makes me think that motor performance (not diagnosis) predicts learning.
If I add age into the model (Learning ~ A + B + age), age is not a significant contributor (Wald = 0.6, p = 0.4), but A and B are significant contributors. That also makes me think motor performance predicts learning.
If I add diagnosis to the model either with age or without age (Learning ~ A + B + age + Dx; Learning ~ A + B + age + Dx), Dx is never a significant contributor. That makes me think that performance predicts learning.
So, if it's A and B that predict learning, and I have the same levels for A and B across my groups when controlling for age, why am I seeing such a difference in the binary outcome for my groups?
Maybe I'm missing an interaction term, or this is simply because linear and logistic regression work differently. What's real?