Maybe it's just that I'm tired, but I'm having trouble trying to understand the Forward Stagewise Regression algorithm. From "Elements of Statistical Learning" page 60:
Forward-stagewise regression (FS) is even more constrained than forward-stepwise regression. It starts like forward-stepwise regression, with an intercept equal to [the mean of] y , and centered predictors with coeﬃcients initially all 0.
At each step the algorithm identiﬁes the variable most correlated with the current residual. It then computes the simple linear regression coeﬃcient of the residual on this chosen variable, and then adds it to the current co- eﬃcient for that variable. This is continued till none of the variables have correlation with the residuals—i.e. the least-squares ﬁt when N > p.
So, is this the algorithm?:
b=mean(y) b[2..n]=0 r=(y-X*b) index, maxCorr = max(transpose(r)*X) while(abs(maxCorr) > someThreshold) b[index]=b[index]+regress(r,X[1..n][index]) r=(y-X*b) index, maxCorr = max(transpose(r)*X)
Where b is a column-vector of the coefficients, X is a matrix of inputs, and y is a column-vector of outputs. I.e. y=X*b+error.
Asking because this algorithm gives me only a few non-zero coefficients on the dataset I'm testing it on (with threshold=.0001), and the prediction accuracy isn't very good at all.