I am reading a book for the lasso-regression. However, I think that this is a more general question regarding the derivative of the OLS-term:

enter image description here

Can someone explain me why this $c_j$ term is measuring the correlation between $x_j$ and the partial residual. I cant see there the structure of the correlation Formula between them


1 Answer 1


Here is first attempt at an answer, as I have come across a similar intuition elsewhere:

In general correlation is defined as

$$\rho_{x,y} = \frac{\sum_i (x_i - \bar x)(y_i - \bar y)}{\sigma_x \sigma_y}$$

That is the covariance divided by a normalizing factor which makes the correlation unit-less

Since the denominator is always positive (standar deviations are positive) we can push them into the proportionality sign, hence the correlation is proportional to

$$ \rho_{x,y}\propto \sum_i (x_i - \bar x)(y_i - \bar y) = \sum_i (x_i - E[x])(y_i - E[y]) $$

Now in this case we have the term $c_j = \sum_i x_{ij}(y_i - w^T_{-j}x_{i,-j}) = \sum_i x_{ij}(y_i - \hat y_{i,-j}) $

Where $\hat y_{i,-j}$ is the predicted value of the model, using all features except feature $j$.

In a OLS regression model, the expectation of the random variable y is $E(Y_i) = \alpha + w x_i$ where $\alpha$ is a constant intercept and $w$ is the slope of the regression line, so again $E(Y_i) \propto wx_i$.

Putting all together we have

$$ c_j = \sum_i (x_{ij} - 0)(y_i - \hat y_{i,-j}) \propto \sum_i (x_{ij} - 0)(y_i - E[y_{i,-j}]) $$

Which would imply / assume that the mean of the $x_{ij}$ is zero (i.e. data is centered)

  • $\begingroup$ In case of the lasso the mean of every feature should be zero because of the centering $\endgroup$
    – rook1996
    Commented Jun 15, 2018 at 19:19
  • $\begingroup$ Well that works out then $\endgroup$ Commented Jun 15, 2018 at 19:33
  • $\begingroup$ Great answer ! Just one question. I didnt understand how you can ommit the variance in that case and argue with the proportional statement. $\endgroup$
    – rook1996
    Commented Jun 15, 2018 at 20:47
  • 1
    $\begingroup$ The standard deviations in the denominator are always positive, so they won't change the sign of the numerator, hence you can "push" them into the proportionality sign - I would be curious to have someone double check though - if glen_b or whuber see this in the future, thanks ! $\endgroup$ Commented Jun 15, 2018 at 22:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.