I am experimenting with some regularized linear regression methods using sklearn and noticed that Ridge does not accept warm start. I found it odd as many other methods do accept like Lasso, LogisticRegression, etc.

Is there a deeper reason for this? Does Ridge not benefit from warm starting?

  • 7
    $\begingroup$ Ridge can be solved as a system of linear equations, so does not need an iterative algorithm. LASSO is not solvable as a linear system, so needs some kind of iterative solver. $\endgroup$ Apr 30, 2018 at 18:29

1 Answer 1


Ridge regression can be solved in one shot as a system of linear equations:

$$ \hat \beta = (X^t X + \lambda I)^{-1} X^t y $$

So ridge regression is usually solved with a linear equation solver, just like linear regression.

For example, sklearn uses the singular value decomposition of the matrix $X$:

$$ X = UDV^{-1} $$

To re-express this system as

$$ \hat \beta = V (D^2 + \lambda I)^{-1} DU^{t}y $$

For a derivation of this equation, see The proof of shrinking coefficients using ridge regression through spectral decomposition.

Notice that this equation is rather nicer than it may seem. The $D^2 + \lambda I$ matrix is diagonal, so inverting it is just taking the reciporical of the entries. Then $(D^2 + \lambda I)^{-1} D$ is also diagonal, and the matrix product is just the product of the diagonal entries.

Here's the source code from sklearn:

def _solve_svd(X, y, alpha):
    U, s, Vt = linalg.svd(X, full_matrices=False)
    idx = s > 1e-15  # same default value as scipy.linalg.pinv
    s_nnz = s[idx][:, np.newaxis]
    UTy = np.dot(U.T, y)
    d = np.zeros((s.size, alpha.size), dtype=X.dtype)
    d[idx] = s_nnz / (s_nnz ** 2 + alpha)
    d_UT_y = d * UTy
    return np.dot(Vt.T, d_UT_y).T

Except for the small amount of gymnastics to deal with the zero singular values, this code lines up exactly with the equation above.

LASSO on the other hand, has no simple expression in terms of linear algebraic operations. For LASSO we need some kind of iterative solver, and so the concept of re-starting the iteration at the previous solutions makes sense.

  • $\begingroup$ Sklearn's ridge implementation also allows for stochastic average gradient descent as the solver. Shouldn't it be possible to do a warm start with that? $\endgroup$ Nov 15, 2018 at 19:39
  • 1
    $\begingroup$ Of course, but you would only need to do so if your data was so large that directly solving the system of linear equations is untenable. $\endgroup$ Nov 15, 2018 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.