2
$\begingroup$

I am estimating a model where the DV is a binary variable and the key independent variable is the interaction between a dummy variable and a continuous variable. I am getting an very odd result where the sign of the coefficient switches between a positive and negative value when I go from a LPM to a Logit model, (though both remain insignificant.)

I've searched all over my data for a coding error, (which my adviser said is the most likely culprit), but I can't find anything amiss. Are there any other conditions that might explain this? I am using robust standard errors in both models.

Thank you for your help! Also apologies if I'm not including enough information, this is my first time with this site.

UPDATE:

As per @Isabella-Ghement's comment in the answer, here are the plots I produced for both the LPM and Logit models she described in the answer.

Actual Data, (black), vs. LPM, (green) vs. Logit, (red) , Dummy=0:

Dummy==0 Actual Data, (black), vs. LPM, (green) vs. Logit, (red) , Dummy=1:

Dummy==1 Update 2:

Here is the qqplot for the LPM:

enter image description here

And for the logit model, (from DHARMa):

enter image description here

$\endgroup$
  • 2
    $\begingroup$ They’re two different estimation methods that will generally yield different results. So it’s not that surprising that they could yield different signs for an irrelevant predictor, for example. $\endgroup$ – The Laconic Mar 22 at 2:04
3
$\begingroup$

There's nothing like plotting your data to get some insights into what might be going on.

For the LPM model (linear probability model), you want to plot your DV versus CONT separately for each of the values of DUMMY, where CONT is the continuous predictor and DUMMY is the dummy predictor variable. When DUMMY = 0, the plot of DV against CONT would look similar to the first plot listed on this page: How to plot binary (presence/absence - 1/0) data against continuous variables. When DUMMY = 1, the plot will change appearance but it will follow the same visual style. For both plots, DV should be plotted on the Y axis and CONT on the X axis.

The LPM model can be stated like:

DV = beta0 + beta1*CONT + beta2*DUMMY + beta3*CONT*DUMMY + epsilon 

After fitting this model to the data, you can obtain the estimated values of the coefficients beta0, beta1, beta2 and beta3 - let's call them b0 through b3. Using this information, you can add a fitted regression line to each of your two plots. That line will depict how the estimated probability that DV = 1 changes with the values of CONT when DUMMY = 0 and when DUMMY = 1, respectively. The end plots should look similar to the plot located in the left panel of the first figure found in the document available at https://are.berkeley.edu/courses/EEP118/fall2010/section/13/Section%2013%20Handout%20Solved.pdf (except the plots will show both the data and the fitted probability line).

The fitted probability lines produced by the LPM model can be obtained by plotting in the corresponding plot of DV versus CONT:

  1. b0 + b1*CONT versus CONT for DUMMY = 0;

  2. (b0 + b2) + (b1 + b3)*CONT versus CONT for DUMMY = 1.

Since your LPM model includes an interaction term, you would expect the two lines to potentially have different slopes.

Now, each of these two plots can be enhanced by adding the corresponding nonlinear fitted probability curve produced by the binary logistic regression model. This model can be stated as:

log (Odds that DV is equal to 1) = gamma0 + gamma1*CONT + gamma2*DUMMY + gamma3*CONT*DUMMY 

and then re-expressed as:

Prob that DV is equal to 1 = exp(gamma0 + gamma1*CONT + gamma2*DUMMY + gamma3*CONT*DUMMY)/
                             [1 + exp(gamma0 + gamma1*CONT + gamma2*DUMMY + gamma3*CONT*DUMMY)]

If you fit the binary logistic model (as expressed in its log odds formulation) to the data and get the estimated values of its coefficients - let's call them g0 through g3 - then you can plot the following fitted probability curves in your plots:

The fitted probability lines produced by the LPM model can be obtained by plotting the following:

  1. exp(g0 + g1*CONT)/[1 + exp(g0 + g1*CONT)] versus CONT for DUMMY = 0;

  2. exp((g0 + g2) + (g1 + g3)*CONT)/[1 + exp((g0 + g2) + (g1 + g3)*CONT)] versus CONT for DUMMY = 1;

in the appropriate plot of DV versus CONT.

The fitted probability curves should look similar to the plot located in the right panel of the first figure found in the document available at https://are.berkeley.edu/courses/EEP118/fall2010/section/13/Section%2013%20Handout%20Solved.pdf (except the plots will also show both the data and the fitted probability line produced by the LPM model).

You should be on the lookout for two types of patterns in your data for each plot (recall that one plot corresponds to DUMMY = 0 and the other to DUMMY = 1).

A. Most of the 0 values for the DV variable are clumped at the lower end of the range of CONT values, and most of the 1 values for the DV variable are clumped at the higher end of the range of CONT values:

DV = 1:          ooooooooooooo

DV = 0:   ooooooooooo

          ---- CONT ---------->

B. Most of the 0 values for the DV variable are clumped at the higher end of the range of CONT values, and most of the 1 values for the DV variable are clumped at the lower end of the range of CONT values:

DV = 1:   ooooooooooooo

DV = 0:             ooooooooooo

          ---- CONT ---------->

In scenario A, you would expect CONT to have a positive linear effect on the probability that Y = 1 for the LPM model and a positive nonlinear effect on the same probability.

In scenario B, you would expect CONT to have a negative linear effect on the probability that Y = 1 for the LPM model and a negative nonlinear effect on the same probability.

In general, I agree with your supervisor that if you encounter patterns such as the ones described in the above scenarios in your plots, you would intuitively expect the fitted probability line produced by LPM and the fitted probability curve produced by the logistic regression model to follow the same direction (e.g., both positive or both negative). If that is not the case, one of the models is not appropriate for the data - most likely, the LPM model.

In any event, plotting your data will reveal if you have any other issues in the DV data that might affect the LPM model fit - for instance, the majority of your DV data values are equal to 0 (or perhaps the majority are equal to 1). Or there could be issues with your CONT variable - too narrow a range of values; only a few distinct values; outliers or gaps in its distribution; etc.

$\endgroup$
  • 1
    $\begingroup$ This is so helpful! Thank you Isabella! $\endgroup$ – StanO Mar 25 at 2:35
  • $\begingroup$ Just a small clarification, did you mean to say that in plot 2 for both the LPM and the Logit, that DUMMY=1, not DUMMY=0? That appears to be the case, but I just wanted to double check. $\endgroup$ – StanO Mar 26 at 2:20
  • $\begingroup$ Good catch! That's exactly what I meant and I edited my answer to reflect that plot 2 for both models should use DUMMY = 1. $\endgroup$ – Isabella Ghement Mar 26 at 3:02
  • $\begingroup$ You can post your plots in your question as an addendum if you want - then we can see better what is going on. $\endgroup$ – Isabella Ghement Mar 26 at 3:09
  • 1
    $\begingroup$ Yeah, when I strip the other controls from the model, the coefficient signs for both models are the same and, as the above plots show, the fits agree. However, when I add other controls, the signs flip and the problems I described above begin. I'm sorry, I should have been more clear on that. $\endgroup$ – StanO Mar 27 at 0:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.