# Comparing logistic regression models

I have a 2(truthfulness) x 2 (immediate test) study where I am looking to compare 2 logistic regression models.

Both are classifying the same outcome variable: truthfulness (truthful versus deceptive -- an experimental manipulation where they were told to tell the truth or lie). The independent variable in both is a score on a delayed test.

The first model uses scores for people who did the immediate test, the other model uses scores for people who did not do the immediate test. I want to know whether truthfulness can be classified better using scores on the delayed test when people have previously done the immediate test.

However, I am confused as to how to compare logistic regression models. Is it possible to do that even though models are estimated from different samples? Is it possible to interpret differences between classification accuracy tables as meaningful? And is it possible to statistically test the differences between models?

-
How well do the two models compare on their validation data sets? –  whuber Sep 15 '11 at 23:37
i'm sorry i don't understand what that means, do you mean how good a fit are the models to the data? –  Anita Sep 15 '11 at 23:39
When you want to know how well a logistic regression classifies things, you need to test it on a set of "validation" data that are separate from, and independent of, the data used to fit the model. (Testing with the same dataset gives invalid, misleading results.) Often this is done by randomly splitting all the data at the outset, fitting the model on one portion, and testing on the other. At this point your question is easily answered: you compare models based on how they perform on the validation data. –  whuber Sep 16 '11 at 14:49
If you are careful in using resampling, you can get accurate results using the original dataset. You have to honestly repeat all analytical steps that involved looking at associations with $Y$. Data splitting is not very reliable unless you have more than 15,000 observations. In other words, if you split the data again, accuracy indexes will vary too much from what you obtained with the first split. To the OP's last question, classification tables do not play much of a role in that setting, being based on a discontinuous improper accuracy scoring rule. –  Frank Harrell Nov 15 '11 at 13:38
@Frank - can you provide references or extended information on this. Most MI/NN profs allow splitting on as few as 100 observations, it is typical in the field. –  EngrStudent May 3 at 20:06
show 1 more comment