1
$\begingroup$

As we know, the gradient of ridge regression is: $$ g = \frac{\partial L}{\partial \theta} = -X_i^T(y_i-X_i\theta)+2\lambda\theta $$ where $X_i$ is the $i$th training sample. The update of $\theta$ is then: $$ \theta^+ =\theta-\eta g $$ with learning rate $\eta$.

My question is: If $\lambda$ is very huge, then the first term in gradient $-X_i^T(y_i-X_i\theta)$ can be ignored, which means lost function cannot be optimized since $g$ is irrelevant to training sample. Am I wrong about this? (The thing is: I tried to use python package to run ridge regression, and the regularization parameter $\lambda$ is a huge value, obtained from validation set. Then I tried to implement stochastic gradient descent (as a comparison), but I found the lost cannot decrease to the lost obtained from python model. Actually, the lost doesn't decrease at all with this huge $\lambda$.)

Improve this question
$\endgroup$
5
  • $\begingroup$ Which python package are you talking about.eg scikit learn uses 'C' parameter which is inverse of regularisation parameter ie large number is no regularisation. Similarly might be scaling factor difference are you minimising sum of errors or mean of errors + regularisation term $\endgroup$
    – seanv507
    Commented Sep 19, 2018 at 8:24
  • $\begingroup$ Yes, I'm using sklearn: reg = linear_model.Ridge(alpha =...), where alpha is set to my regularization parameter $\lambda$. $\endgroup$
    – Data G
    Commented Sep 19, 2018 at 14:18
  • $\begingroup$ ok so your alpha is matching (as opposed to C in scikit learn logistic regression), but I think your gradient is not matching: you should be dividing alpha by the number of samples to match scikit learn. since you are doing the update for each sample. eg in your g term you should have a sum over the X_i term, ie your sgd $\lambda=\alpha/N\_samples$ $\endgroup$
    – seanv507
    Commented Sep 19, 2018 at 14:21
  • $\begingroup$ Thank you!@seanv507. It does work. (The lost is still somewhat higher than the python package model, though. But I guess it's just the intrinsic shortcoming of sgd?) $\endgroup$
    – Data G
    Commented Sep 19, 2018 at 15:46
  • $\begingroup$ You might have to reduce the learning rate?? The learning rate has to be 'smaller' than the maximum curvature or you will oscillate around minimum $\endgroup$
    – seanv507
    Commented Sep 19, 2018 at 17:25

1 Answer 1

Reset to default
1
$\begingroup$

Ridge Regression python package has several solver options, and is not employing the same method as you. Your implementation is the very basic of gradient descent method that employs constant learning coefficient I presume, i.e. you don't have any strategy for adaptively setting your learning coefficient. And in sensitive cases as yours (i.e. large numbers), this can easily lead to different results. Library methods, in general, are products of highly experienced researchers and developers and highly stable in cases of numerical challenges.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you! So, with a huge regularization parameter, sgd may not work at all? Is that true? $\endgroup$
    – Data G
    Commented Sep 19, 2018 at 14:08
  • $\begingroup$ Well, blaming the algorithm is wrong. A correct choice of learning constant can still lead to correct solution, but finding it is not easy. Also, it might be so small that you might not be able to use it in a stable way, because we're using finite precision machines. $\endgroup$
    – gunes
    Commented Sep 19, 2018 at 14:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.