I understand the math behind the coordinate descent Algorithm for LS and lasso. my question is related to CD plus Karush-Kuhn-Tucker (KKT) Conditions. So,
from Coordinate descent soft-thresholding update operator for LASSO
\begin{aligned} \frac{\partial }{\partial \theta_j} RSS^{OLS}(\theta) & = - \rho_j + \theta_j z_j \end{aligned}
so, we can say that \begin{aligned} θ_j=ρ_j \end{aligned}
and by definiton \begin{aligned} ρ_j= \sum_{i=1}^m x_j^{(i)} \left[y^{(i)} - \sum_{k \neq j}^n \theta_k x_k^{(i)} \right] \end{aligned}
so, they explain the idea behind $\rho$ as follows
Well what $\rho_j$ is doing is it's a measure of the correlation between our future $j$ and this residual between the predicted value. Where remember that prediction was formed excluding that $j$th feature from the model and the true observation. So we're looking at our prediction, the difference relative to the truth, when $j$ is not included in the model. And then we're seeing how correlated is that with the $j$th feature.
My questions:
1- when they say the term correlation, what do they mean in specific? Pearson correlation or what?. if I multiplied the values for a feature by the residuals from that model by which that feature was excluded from that model, I get a correlation measure.? ( i have never heard of this before!)
2 - for example let's say we have 3 features, let's jump to the second iteration for CD algorithm, and now we want to test the second feature, and now we're computing rho, we will multiply the values for the second feature by the residuals from the model which that feature was excluded? and test the correlation between them.? Sorry but I didn't get the general idea behind it
