Correlation between the j-feature and partial residual I am reading a book for the lasso-regression. However, I think that this is a more general question regarding the derivative of the OLS-term:
 
Can someone explain me why this $c_j$ term is measuring the correlation between $x_j$ and the partial residual. I cant see there the structure of the correlation Formula between them
 A: Here is first attempt at an answer, as I have come across a similar intuition elsewhere: 
In general correlation is defined as 
$$\rho_{x,y} = \frac{\sum_i (x_i - \bar x)(y_i - \bar y)}{\sigma_x \sigma_y}$$
That is the covariance divided by a normalizing factor which makes the correlation unit-less 
Since the denominator is always positive (standar deviations are positive) we can push them into the proportionality sign, hence the correlation is proportional to 
$$ \rho_{x,y}\propto \sum_i (x_i - \bar x)(y_i - \bar y) = \sum_i (x_i - E[x])(y_i - E[y]) $$
Now in this case we have the term $c_j = \sum_i x_{ij}(y_i - w^T_{-j}x_{i,-j}) = \sum_i x_{ij}(y_i - \hat y_{i,-j}) $
Where $\hat y_{i,-j}$ is the predicted value of the model, using all features except feature $j$.
In a OLS regression model, the expectation of the random variable y is $E(Y_i) = \alpha + w x_i$ where $\alpha$ is a constant intercept and $w$ is the slope of the regression line, so again $E(Y_i) \propto wx_i$. 
Putting all together we have 
$$ c_j = \sum_i (x_{ij} - 0)(y_i - \hat y_{i,-j}) \propto  \sum_i (x_{ij} - 0)(y_i - E[y_{i,-j}]) $$
Which would imply / assume that the mean of the $x_{ij}$ is zero (i.e. data is centered)  
