Unlike linear regression, we do not have close-form result for the weights when training a logistic regression model. So I learned that we can either apply Newton's method or (stochastic) gradient descent when minimizing the binary cross entropy loss together with l1/l2 regularization.

E.g., if we use Lasso, the negative log likelihood becomes $$NLL(w) = - log P(Y|X,w) - \lambda \sum_{i = 1}^{K} w_i ^2$$

If we use gradient descent to find the global minimum, for a specific $\lambda$, we will get one curve for training loss and one curve for validation loss like the pic shown below (I plotted curves for 2 different $\lambda$) enter image description here

Then how do we select the best $\lambda$? Do we still do N-fold cross validation? I guess yes? Then do we compare loss at the two 'X' points where validation loss is minimized or do we compare loss at two 'O' points where training loss is minimized? From my view, I prefer the first scenario to compare the 2 crosses but I don't know what is actually implemented in scikit learn or other packages? (For Newton's method, it is very fast to converge to the minimum so we will not have the smooth curve plotted here.)

  • $\begingroup$ Whether you use SGD or Newton-Raphson or any exact method is not directly related to the choice of $\lambda$, which is simply seen as a constant. When dealing with a convex function, any exact method is guaranteed to yield the global optimum. $\lambda$ should be set in a separate hyperparameter optimisation stage where you will obviously be comparing the two minima between val1 and val2 in your plots (I don't see why you should care about the training loss). $\endgroup$ – Digio Dec 7 '17 at 10:22
  • $\begingroup$ Thanks! I agree with you when I first thought about it. But if we really look at the training curve and select the minimized validation loss (which we will use to compare the select the best $lambda$), we are potentially preventing overfitting at the same time. I also concern that in some cases, 'x' may select $\lambda_1$ whereas 'o' may select $\lambda_2$. $\endgroup$ – Sophie Dec 8 '17 at 2:29
  • $\begingroup$ We're saying the same thing, it's the validation loss that should be taken into account in order to avoid overfitting. $\endgroup$ – Digio Dec 8 '17 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.