2
$\begingroup$

I have been running logit regressions on large samples in order of hundreds of thousands in a preliminary study that will eventually end up in non-parametric tests. Both my dependent variable and my independent variable of interest are binary for now. I started with a very underspecified model containing only my IV of interest in a single logit regression. Since the dataset is quite large, I haven't been putting much trust into the significance level of my coefficients which seem to be almost always perfect. Instead, I have been calculating odds ratios that can give me some kind of indication of the predictive power of the model. While I was running my underspecified regression (glm(dv ~ iv, family = binomial(link = logit)) on preliminary noisy data the odds ratios seemed to be at an acceptable level, so I decided to proceed to the next step, clean the data and import the control variables, etc...

Now the issue is: since I have started using the full clean data with real control variables, the odds ratios of my variable of interest have started exploding.

Consider this:

glm(clean_dv ~ clean_iv, family = binomial(link = logit))

clean_iv coefficient: 4.619625

clean_iv stderr: 0.267083

clean_iv odds: 101.45602

glm(clean_dv ~ clean_iv+noisy_cv1+noisy_cv2, family = binomial(link = logit))

clean_iv coefficient: 6.233e+00

clean_iv stderr: 2.727e-01

clean_iv odds: 509.3612309

glm(clean_dv ~ clean_iv+clean_cv1+clean_cv2, family = binomial(link = logit))

clean_iv coefficient: 5.582e+00

clean_iv stderr: 2.369e-01

clean_iv odds: 265.6611359

The control variables behave just normally. They are significant and have acceptable odds ratios.

Better odds ratios should supposedly be good news, but at this level I don't know how to interpret them anymore. Any help is appreciated.

Improve this question
$\endgroup$
8
  • $\begingroup$ Since logistic regression requires listwise deletion (complete case analysis), chances are your inclusion of control variables has markedly reduced the sample size, in a way that biases the iv's coefficient upwards. $\endgroup$
    – rolando2
    Commented Jan 15, 2015 at 3:44
  • 2
    $\begingroup$ Do you perhaps have complete separation anywhere? $\endgroup$
    – Glen_b
    Commented Jan 15, 2015 at 7:36
  • 2
    $\begingroup$ This looks more like a stats question, even if there is some mention of stata and R. The main question is stats, shoiuld be open. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jan 15, 2015 at 8:21
  • 1
    $\begingroup$ First step: look at the sample size used by your model and compare that with the sample size in your data. Second step: just stare at cross-tabulations: look for (nearly) empty cells, figure out why those cells became empty due to datacleaning (i.e. did you "overclean" your data). Third step, revisist every datacleaning step and every control variable till you find the answer. That is a lot of work, but that is normal: preparing your data is usually by far the most time consuming part of a statistical analysis. $\endgroup$
    – Maarten Buis
    Commented Jan 15, 2015 at 8:51
  • 1
    $\begingroup$ My first step would be to think about that cell in substantive terms: Does it make sense that it is almost empty, do I want to compare that cell with others, which comparisons interest me most. Sometimes that is enough to find a solution. Only after that, I would start to consider more "technical" solutions. $\endgroup$
    – Maarten Buis
    Commented Jan 15, 2015 at 12:06

1 Answer 1

Reset to default
1
$\begingroup$

You can do a formal check for separation using the safeBinaryRegression package: Konis (2007), "Linear programming algorithms for detecting separated data in binary logistic regression models", DPhil, U. Oxf. But the Wald standard errors don't seem suspiciously large; & whether the odds ratio estimate is sensible should be evident from a cross-tabulation. A large odds ratio per se doesn't indicate any problem in the fitting procedure or present difficulties of interpretation. If data cleaning made a huge difference we can't guess why unless you tell us more about what it involved.

Improve this answer
$\endgroup$
1
  • $\begingroup$ By "cleaning" I didn't mean removing any data or getting rid of incomplete data. My cleaning simply involved replacing some crude measurements, mostly approximations (that is real data + some noise), with accurate measurements. For instance I had used the date of birth as a control, but then I calculated the real age of the person at the time of observation (which differed by 1 to 3 years for different observations). And, yes, the errors are correct. Actually all tests come out just right. But I will try the safeBinary as well. $\endgroup$
    – retrography
    Commented Jan 15, 2015 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.