# stochastic gradient descent of ridge regression when regularization parameter is very big

As we know, the gradient of ridge regression is: $$g = \frac{\partial L}{\partial \theta} = -X_i^T(y_i-X_i\theta)+2\lambda\theta$$ where $X_i$ is the $i$th training sample. The update of $\theta$ is then: $$\theta^+ =\theta-\eta g$$ with learning rate $\eta$.

My question is: If $\lambda$ is very huge, then the first term in gradient $-X_i^T(y_i-X_i\theta)$ can be ignored, which means lost function cannot be optimized since $g$ is irrelevant to training sample. Am I wrong about this? (The thing is: I tried to use python package to run ridge regression, and the regularization parameter $\lambda$ is a huge value, obtained from validation set. Then I tried to implement stochastic gradient descent (as a comparison), but I found the lost cannot decrease to the lost obtained from python model. Actually, the lost doesn't decrease at all with this huge $\lambda$.)

• Which python package are you talking about.eg scikit learn uses 'C' parameter which is inverse of regularisation parameter ie large number is no regularisation. Similarly might be scaling factor difference are you minimising sum of errors or mean of errors + regularisation term – seanv507 Sep 19 '18 at 8:24
• Yes, I'm using sklearn: reg = linear_model.Ridge(alpha =...), where alpha is set to my regularization parameter $\lambda$. – Data G Sep 19 '18 at 14:18
• ok so your alpha is matching (as opposed to C in scikit learn logistic regression), but I think your gradient is not matching: you should be dividing alpha by the number of samples to match scikit learn. since you are doing the update for each sample. eg in your g term you should have a sum over the X_i term, ie your sgd $\lambda=\alpha/N\_samples$ – seanv507 Sep 19 '18 at 14:21
• Thank you!@seanv507. It does work. (The lost is still somewhat higher than the python package model, though. But I guess it's just the intrinsic shortcoming of sgd?) – Data G Sep 19 '18 at 15:46
• You might have to reduce the learning rate?? The learning rate has to be 'smaller' than the maximum curvature or you will oscillate around minimum – seanv507 Sep 19 '18 at 17:25