Orthogonal polynomials + cross validation: should subsetting be done prior or after constructing the orthogonal polynomials?

Question

So, just to start... I've just learned of orthogonal polynomial regression today. I've gone through the master's-level linear models courses, and we did not cover that topic. I was always under the assumption that, especially for polynomial regression, that $\mathbf{X}^{T}\mathbf{X}$ is invertible most of the time, and then you just get the coefficients from $\hat{\boldsymbol\beta} = (\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}^{T}\mathbf{y}$, and everything's great. Any explanation of what's going on here given my background on this would be appreciated as well on the side.

In a question on StackOverflow, I had noticed that different results were obtained under these two scenarios in R:

1) If I had done a regression using glm() with the data argument the training subset of the data and the subset argument omitted, the data are filtered first to only use the training subset of data, and secondly, the orthogonal polynomials are constructed.

2) If I had done a regression using glm() with the data argument the entire data set (training + test data) and the subset argument equal to the (row indices of) the training subset of data, the orthogonal polynomials are constructed first, and secondly, the data are subsetted.

I wanted to call attention to this, as I couldn't find any guidance behind this in Google searching.

For the purpose of cross validation, which one of the two scenarios above should be done? Does it even matter? One of the commenters on the StackOverflow question I posted above pointed out that the fitted values are still the same (according to the GLM fit, that is). However, I can see issues with interpretation of parameter estimates.

FYI: Introduction to Statistical Learning uses the second approach in its R lab examples.

Answer 1

For the purposes of cross-validation or any other resampling scheme, I would strongly suggest you leave the test/validation data completely out of the training routine. Section 4.4 "Resampling Techniques" from Kuhn and Johnson's Applied Predictive Modeling has a nice and concise exposition of the matter; section 5.3 "Model Validation" from Harrell's Regression Modeling Strategies gives a more academic exposition. (Both books are excellent reads.)

Any sample-derived measurements used during model fitting/training should be using information that is only available prior to prediction. Otherwise we have data-leakage, a situation which can easily manifest to poor generalisation of our model. For the case you outline, using information only available prior to prediction means that you do the subsetting of your data prior to constructing the orthogonal polynomial basis. Indeed, there will be cases where the effect of using the whole model can be minimal (eg. the shape of a basis usually will not change substantially given a large enough subset), but they will be other cases where the effect of partitioning can be severe (eg. when normalising a leptokurtotic variable and trying to compute standard deviations). Therefore, I suggest opting for the safe choice and do the sub-setting first.