I am a bit confused about the formulation of the maximum liklihood equation for logistic regression for ridge regression (and similar for lasso regression).
Where andrew Ng (coursera course) states the following
$\frac{1}{N}$loglik$(\beta)-\frac{\lambda}{2N}\sum^{p}_{i=1}\theta^2 $ --> EQ1
I also found the next one in "regularization paths for GLM via coordinate descent - eq 13"
$\frac{1}{N}$loglik$(\beta)-\frac{\lambda}{2}\sum^{p}_{i=1}\theta^2 $ --> EQ2
And yet again in "An introduction to statistical learning - eq 6.5"
loglik$(\beta)-\lambda\sum^{p}_{i=1}\theta^2 $ --> EQ3
So the main part is always similar but the denominators change. I understand why the division by 2 comes into play (due to the easier calculations when taking the derivative). From reading up on this forum, I also understand why it is better to divide by N (in short, due to the solution becoming independent of the amount of observations (In particular for large data sets)).
The issue I have is that, I want to compare a matlab model using the lassoglm command - model 1 - with a ridge regression model based upon the first equation from Andrew Ng - model 2.
If I set the alpha in model 1 to ~0 (and thus converting it effectively into a ridge regression model) I get a certain deviance. Which, for the same lambda and CV options, should match the deviance I get from model 2. Sadly this is not the case, and the deviances only match each other when I change the model 2 max liklihood equation from EQ 1 to EQ 2.
So I think my questions are
- Can I conclude that the lassoglm uses EQ2 to fit the model. And if so, why is that the case as the arguments are in favor for using EQ1.
- whether it makes sense to compare deviances with each other from models that use different max liklihood functions.
Thanks,
