# Orthogonal polynomials + cross validation: should subsetting be done prior or after constructing the orthogonal polynomials?

So, just to start... I've just learned of orthogonal polynomial regression today. I've gone through the master's-level linear models courses, and we did not cover that topic. I was always under the assumption that, especially for polynomial regression, that $\mathbf{X}^{T}\mathbf{X}$ is invertible most of the time, and then you just get the coefficients from $\hat{\boldsymbol\beta} = (\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}^{T}\mathbf{y}$, and everything's great. Any explanation of what's going on here given my background on this would be appreciated as well on the side.

In a question on StackOverflow, I had noticed that different results were obtained under these two scenarios in R:

1) If I had done a regression using glm() with the data argument the training subset of the data and the subset argument omitted, the data are filtered first to only use the training subset of data, and secondly, the orthogonal polynomials are constructed.

2) If I had done a regression using glm() with the data argument the entire data set (training + test data) and the subset argument equal to the (row indices of) the training subset of data, the orthogonal polynomials are constructed first, and secondly, the data are subsetted.

I wanted to call attention to this, as I couldn't find any guidance behind this in Google searching.

For the purpose of cross validation, which one of the two scenarios above should be done? Does it even matter? One of the commenters on the StackOverflow question I posted above pointed out that the fitted values are still the same (according to the GLM fit, that is). However, I can see issues with interpretation of parameter estimates.

FYI: Introduction to Statistical Learning uses the second approach in its R lab examples.

• If the verification data are involved in your modeling in any way whatsoever then they're not honest verification data. In some sense the distinction between (1) and (2) doesn't matter because predictions are unaffected (and it's likely that if you do any cross-validation on the training data you won't be recomputing the orthogonal polynomials for each fold, anyway, so you're already sort of relying on this not mattering). – whuber Oct 30 '17 at 20:52
• For the purposes of cross-validation or any other scheme, I would strongly suggest you leave the test/validation data completely out of the training routine. That means you do the subsetting prior to your constructing the orthogonal polynomial. Yes, there will be cases where the effect can be minimal (eg. the shape of a basis usually won't change horribly given a large enough subset), but they will be other cases where the effect can be severe (eg. when normalising a leptokurtotic variable and trying to compute std. deviations). Opt for the safe choice and do the sub-setting first. – usεr11852 Oct 30 '17 at 23:59
• These comments could just as well be answers IM(very)HO. – eric_kernfeld Nov 2 '17 at 16:31
• @Clarinetist: OK! I will flesh it out a bit more in a few hours. – usεr11852 Nov 2 '17 at 20:01