Suppose I have an $N×1$ vector $Y_{in}$ of response values an $N×P$ matrix of predictors $X_{in}$ whose individual columns exhibit significant correlation. Let's further suppose that this matrix $X$ represents in-sample data for a predictive model; i.e., I wish to train on $X_{in}$ and then predict on out-of-sample data $X_{out}$. Let's suppose further still that for a variety of reasons (which I will not go into here because the value/need for this process is a separate question and not one I wish to explore at this moment) I find it practical to train the model on data whose columns are orthonormal. Then there will be some centering, scaling, and orthogonalization parameters determined from $X_{in}$ that will be later used to transform $X_{out}$ so that, in expectation, the joint distribution of its columns is consistent with orthonormality. Explicitly, I think I'd need center (mean), scale (standard deviation), and a coordinate rotation matrix (from some form of SVD).

Surely, the values in all of those objects (centers, scales, rotation matrix elements) themselves have some variance and this variance will just as surely influence the predictions. My question is: is there any literature, established best practices, or even just ideas of how the parameter estimates in the center, scale, and rotation matrices can be estimated and/or regularized in order to minimize expected out of sample error?

Partial answers are totally acceptable; i.e., it occurred to me that the Stein estimator could be used to regularize the centers. And perhaps use of median absolute deviation in place of standard deviation for the scales. And maybe even an L2 penalty on the rotation matrix elements would work. But I haven't found any serious discussion on regularization during data "preprocessing" so I'm wondering if I'm just missing something or if there's some reason it's not "a thing."

I'm primarily interested in regularization of the rotation matrix, so if an answer provides a reasonable approach to this process I'll likely accept it even if it doesn't discuss centering/scaling, which seem much more straightforward.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.