0
$\begingroup$

Sorry my dataset is pretty big so that I can not post it here.

I have a simple glmnet model in R

coef(glmnet(Y ~ A + B+ C, data = test_data, family = "binomial", intercept = F), s=0)

shows if intercept = F, A will get a negative coefficient while if intercept is T or omitted as T is the default, A will get a positive coefficient. I thought intercept determines where the odd ratio is when all predictors are zero. However, the direction of a predictor should not be changed by intercept... How come? Can someone help to explain?

$\endgroup$
4
  • 1
    $\begingroup$ Is there any specific element of GLM in your question, or do you also wonder about how can the inclusion of the intercept change the sign of the predictor coefficients in simple linear regression? $\endgroup$
    – Kuku
    Oct 28, 2020 at 19:49
  • $\begingroup$ Hi, I wonder how can the inclusion of the intercept change the sign of one of three predictors coefficients in logistic regression. $\endgroup$ Oct 28, 2020 at 20:27
  • 1
    $\begingroup$ Yes, but as to further guide the answer, can you tell me whether you understand how can the inclusion of the intercept change the sign of predictor coefficients in simple linear regression? $\endgroup$
    – Kuku
    Oct 28, 2020 at 21:35
  • $\begingroup$ Do you understand how a coefficient will be biased if you omit the intercept when it should be included in a basic linear model? $\endgroup$
    – Caio C.
    Oct 28, 2020 at 22:24

1 Answer 1

0
$\begingroup$

Glmnet uses a version of Maximum Likelihood Estimation, and for that to work you have to assume the distribution of your data. Notice that in the case of the Logit model:

Let $X = (X_1, \dots , X_k)$, $\alpha = intercept$ and $\beta = (\beta_1, \dots , \beta_k)$:

Without intercept, your error term's distribution would be:

$$G(X \beta) \sim \Lambda(X \beta) = \frac{e^{X \beta}}{1+e^{X \beta}}$$ And your estimators would be: $$\hat{\beta}_{MLE} = argmax \frac{1}{N} \sum (y_i log(\Lambda(X \beta) + (1-y_i)log(1- \Lambda(X \beta))$$

With intercept, it would be:

$$G(X \beta + \beta_0 \alpha) \sim \Lambda(X \beta + \beta_0 \alpha) = \frac{e^{X \beta + \beta_0 \alpha}}{1+e^{X \beta + \beta_0 \alpha}}$$ And your estimators would be: $$\hat{\beta^*}_{MLE} = argmax \frac{1}{N} \sum (y_i log(\Lambda(X \beta + \beta_0 \alpha) + (1-y_i)log(1- \Lambda(X \beta + \beta_0 \alpha))$$

Notice that the FOCs will be nonlinear and the presence or absence of $\alpha$ will change the value of the estimated $\beta$ even for $\beta_i \neq \beta_0$. So, there is no reason to believe that the direction would be the same.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.