I have a 2(truthfulness) x 2 (immediate test) study where I am looking to compare 2 logistic regression models.

Both are classifying the same outcome variable: truthfulness (truthful versus deceptive -- an experimental manipulation where they were told to tell the truth or lie). The independent variable in both is a score on a delayed test.

The first model uses scores for people who did the immediate test, the other model uses scores for people who did not do the immediate test. I want to know whether truthfulness can be classified better using scores on the delayed test when people have previously done the immediate test.

However, I am confused as to how to compare logistic regression models. Is it possible to do that even though models are estimated from different samples? Is it possible to interpret differences between classification accuracy tables as meaningful? And is it possible to statistically test the differences between models?

  • $\begingroup$ How well do the two models compare on their validation data sets? $\endgroup$ – whuber Sep 15 '11 at 23:37
  • $\begingroup$ i'm sorry i don't understand what that means, do you mean how good a fit are the models to the data? $\endgroup$ – Anita Sep 15 '11 at 23:39
  • 3
    $\begingroup$ When you want to know how well a logistic regression classifies things, you need to test it on a set of "validation" data that are separate from, and independent of, the data used to fit the model. (Testing with the same dataset gives invalid, misleading results.) Often this is done by randomly splitting all the data at the outset, fitting the model on one portion, and testing on the other. At this point your question is easily answered: you compare models based on how they perform on the validation data. $\endgroup$ – whuber Sep 16 '11 at 14:49
  • 2
    $\begingroup$ If you are careful in using resampling, you can get accurate results using the original dataset. You have to honestly repeat all analytical steps that involved looking at associations with $Y$. Data splitting is not very reliable unless you have more than 15,000 observations. In other words, if you split the data again, accuracy indexes will vary too much from what you obtained with the first split. To the OP's last question, classification tables do not play much of a role in that setting, being based on a discontinuous improper accuracy scoring rule. $\endgroup$ – Frank Harrell Nov 15 '11 at 13:38
  • $\begingroup$ @Frank - can you provide references or extended information on this. Most MI/NN profs allow splitting on as few as 100 observations, it is typical in the field. $\endgroup$ – EngrStudent - Reinstate Monica May 3 '13 at 20:06

I am confused. You say the DV is the experimental condition? This cannot be. The thing you manipulate is always an independent variable; and score on the test should be the dependent variable.

Given that you have 2x2 tables (apparently 2 of them) you could look at percent correct in each, or sensitivity and specificity in each, and compare these. Testing the difference between two proportions is standard stuff.

Are the two groups comparable? I don't know. It depends on how you got them. Are they both random samples from the same population?

  • $\begingroup$ well no, i know it's the iv in my experiment, it's ok. but i am entering it as the outcome (dv) variable in the logistic regression because i am trying to classify it. yes, the groups are comparable. they are both random samples from 1 population.oh and how would i compare the proportions? a t test? $\endgroup$ – Anita Sep 16 '11 at 0:08
  • $\begingroup$ A t-test is the usual method for comparing two proportions. $\endgroup$ – Peter Flom - Reinstate Monica Sep 16 '11 at 9:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.