0
$\begingroup$

I am a bit confused about the formulation of the maximum liklihood equation for logistic regression for ridge regression (and similar for lasso regression).

Where andrew Ng (coursera course) states the following

$\frac{1}{N}$loglik$(\beta)-\frac{\lambda}{2N}\sum^{p}_{i=1}\theta^2 $ --> EQ1

I also found the next one in "regularization paths for GLM via coordinate descent - eq 13"

$\frac{1}{N}$loglik$(\beta)-\frac{\lambda}{2}\sum^{p}_{i=1}\theta^2 $ --> EQ2

And yet again in "An introduction to statistical learning - eq 6.5"

loglik$(\beta)-\lambda\sum^{p}_{i=1}\theta^2 $ --> EQ3

So the main part is always similar but the denominators change. I understand why the division by 2 comes into play (due to the easier calculations when taking the derivative). From reading up on this forum, I also understand why it is better to divide by N (in short, due to the solution becoming independent of the amount of observations (In particular for large data sets)).

The issue I have is that, I want to compare a matlab model using the lassoglm command - model 1 - with a ridge regression model based upon the first equation from Andrew Ng - model 2.

If I set the alpha in model 1 to ~0 (and thus converting it effectively into a ridge regression model) I get a certain deviance. Which, for the same lambda and CV options, should match the deviance I get from model 2. Sadly this is not the case, and the deviances only match each other when I change the model 2 max liklihood equation from EQ 1 to EQ 2.

So I think my questions are

  • Can I conclude that the lassoglm uses EQ2 to fit the model. And if so, why is that the case as the arguments are in favor for using EQ1.
  • whether it makes sense to compare deviances with each other from models that use different max liklihood functions.

Thanks,

$\endgroup$
  • $\begingroup$ Your trouble comes about because of that "same" in "Which, for the same lambda and CV options...". The symbol $\lambda$ may be the same, but the optimal value is not. If you use CV to select $\lambda$, you should get similar parameter estimates across the three expressions, albeit widely differing $\lambda$s. $\endgroup$ – jbowman Apr 16 '19 at 19:00
  • $\begingroup$ Thanks, that clarifies a lot. $\endgroup$ – Arjan Dexters Apr 18 '19 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.