# Correlation between the j-feature and partial residual

I am reading a book for the lasso-regression. However, I think that this is a more general question regarding the derivative of the OLS-term:

Can someone explain me why this $c_j$ term is measuring the correlation between $x_j$ and the partial residual. I cant see there the structure of the correlation Formula between them

Here is first attempt at an answer, as I have come across a similar intuition elsewhere:

In general correlation is defined as

$$\rho_{x,y} = \frac{\sum_i (x_i - \bar x)(y_i - \bar y)}{\sigma_x \sigma_y}$$

That is the covariance divided by a normalizing factor which makes the correlation unit-less

Since the denominator is always positive (standar deviations are positive) we can push them into the proportionality sign, hence the correlation is proportional to

$$\rho_{x,y}\propto \sum_i (x_i - \bar x)(y_i - \bar y) = \sum_i (x_i - E[x])(y_i - E[y])$$

Now in this case we have the term $c_j = \sum_i x_{ij}(y_i - w^T_{-j}x_{i,-j}) = \sum_i x_{ij}(y_i - \hat y_{i,-j})$

Where $\hat y_{i,-j}$ is the predicted value of the model, using all features except feature $j$.

In a OLS regression model, the expectation of the random variable y is $E(Y_i) = \alpha + w x_i$ where $\alpha$ is a constant intercept and $w$ is the slope of the regression line, so again $E(Y_i) \propto wx_i$.

Putting all together we have

$$c_j = \sum_i (x_{ij} - 0)(y_i - \hat y_{i,-j}) \propto \sum_i (x_{ij} - 0)(y_i - E[y_{i,-j}])$$

Which would imply / assume that the mean of the $x_{ij}$ is zero (i.e. data is centered)

• In case of the lasso the mean of every feature should be zero because of the centering Commented Jun 15, 2018 at 19:19
• Well that works out then Commented Jun 15, 2018 at 19:33
• Great answer ! Just one question. I didnt understand how you can ommit the variance in that case and argue with the proportional statement. Commented Jun 15, 2018 at 20:47
• The standard deviations in the denominator are always positive, so they won't change the sign of the numerator, hence you can "push" them into the proportionality sign - I would be curious to have someone double check though - if glen_b or whuber see this in the future, thanks ! Commented Jun 15, 2018 at 22:33