Applying de-standardised ridge regression coefficients to new test data - how to best handle the mean of y_test?

Question

this is my first post on Stackexchange, so please correct me in any way if I am doing it wrongly.

I just stumbled across this question, I was battling with the same issue, but the posts there greatly clarified things. My question picks up a Slack from that question's discussion.

In an answer to the last comment, the OP asked:

Ok. The coefficients obtained for original data and normalized data are very different. Now if normalization is a recommended procedure, how do i normalize a new test data for whom i have to predict y. Do I have to use the same mean and SD from the training data or they are to be computed from the test data itself? And if the coefficients from the normalized data are used, the predictions are completely out of scale. How do we tackle this issue?

I wanted to pick up that slack. I am aware of the correct answer that was given in response to this question (de-standardise standardised coefficients which were derived from standardised training data and apply those to the new incoming test data to get predictions).

However, I was wondering what you guys think about how to handle y (i.e, the outcome, not the predictors X) of the test data in terms of its mean? More precisely: after having fit my Ridge regression to training data and de-standardised the resulting coefficients including the intercept, I could happily apply that to test data.

But, even if my fitted gradient betas (i.e., the non-intercept betas) reflect well the trend in the test data, it could be that the intercept ('mean') of y in the test data is very different to the mean of y in the training data (= the intercept of the ridge model). If I judge the fit of my ridge model using R^2, the fit would be bad - but solely because of intercept differences.

Therefore, I was wondering whether it makes sense to center y_test, either with its own mean or with the mean of y_train?

Does anyone have any thoughts on this?

Answer 1

You have to view test data as a stand in for a feed of "new" data that your model will receive when used to make decisions in a production setting.

it could be that the intercept ('mean') of y in the test data is very different to the mean of y in the training data (= the intercept of the ridge model). If I judge the fit of my ridge model using R^2, the fit would be bad - but solely because of intercept differences.

Yup, and there's nothing you can or should do about this. Unless you have an active process to update the intercept of your model in production (this is the case is some actuarial applications, for example), this is just the reality of your situation. If the overall mean of the test data drifts from your training mean, that's a real effect that your model will have to deal with when making real life decisions, and you need to fairly evaluate your model's capacity to adapt to that reality.

$R^2$ is not intended as a measure of predictive performance for exactly this reason. Computing $R^2$ on a test data set requires computing the mean of the target in the test data set, and this should be viewed as problematic(*). Indeed, in many production settings, models see one observation at a time (i.e., they are not scored in "batch mode"), so you don't really have anything to take the mean of!

(*) I'm aware this is the default in sklearn. This is a design error in the library, the default should be MSE.