So, just to start... I've just learned of orthogonal polynomial regression today. I've gone through the master's-level linear models courses, and we did not cover that topic. I was always under the assumption that, especially for polynomial regression, that $\mathbf{X}^{T}\mathbf{X}$ is invertible most of the time, and then you just get the coefficients from $\hat{\boldsymbol\beta} = (\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}^{T}\mathbf{y}$, and everything's great. Any explanation of what's going on here given my background on this would be appreciated as well on the side.
In a question on StackOverflow, I had noticed that different results were obtained under these two scenarios in R
:
1) If I had done a regression using glm()
with the data
argument the training subset of the data and the subset
argument omitted, the data are filtered first to only use the training subset of data, and secondly, the orthogonal polynomials are constructed.
2) If I had done a regression using glm()
with the data
argument the entire data set (training + test data) and the subset
argument equal to the (row indices of) the training subset of data, the orthogonal polynomials are constructed first, and secondly, the data are subsetted.
I wanted to call attention to this, as I couldn't find any guidance behind this in Google searching.
For the purpose of cross validation, which one of the two scenarios above should be done? Does it even matter? One of the commenters on the StackOverflow question I posted above pointed out that the fitted values are still the same (according to the GLM fit, that is). However, I can see issues with interpretation of parameter estimates.
FYI: Introduction to Statistical Learning uses the second approach in its R
lab examples.