Geometrical interpretation of correlation of a variable and the residual

Question

I came across this blog post about least angle regression, and at a point he says:

• Find the variable $x_1$ most correlated with the residual. (Note that the variable most correlated with the residual is equivalently the one that makes the least angle with the residual, whence the name.)

• Move in the direction of this variable until some other variable $x_2$ is just as correlated.

But I couldn't visualize it. By "residual" he is referring to the vectors minus the average of the vectors, am I right? What could be a geometrical interpretation of "variable correlation with the residual"? What would be a, let's say, 2D example of this interpretation?

Answer 1

When two variables are highly correlated it means that when you do a scatter plot of the observed pairs they will fall close to a straight line. It is the same here except that one of the variables is a residual. In two dimensions looking at vectors suppose you move from the point (Xmean, Ymean) to a point (Xmean + D, Ymean + E) That vector will have slope E/D. Now if the residual vector at (Xmean + D, Ymean + E) has a change in X and Y similar to the change for the variable X1 being considered it will be highly correlated with X1.