1
$\begingroup$

Imagine I have a large dataset of many variables and many observations. I would like to create a regression model to predict the values of new data. For the sake of ease, say I find a ridge regression model with some value $\lambda$ serves my needs best.
For my model, I decided to center and scale (standardize) my data before fitting the model. I then divide my dataset into training values and testing values, the model is built on the training values. I then test my model using the testing values and I am satisfied with the outcome.

Now, I observe a new observation and I don't know what the true outcome is, I decide to run it through the model I made. Before I feed the values into the model, I should 'standardize' this observation, correct? I.e. shift it by the mean of the data it was built on and then divide it by the standard deviation (not sure if this is correct, it is part of my question). Then I feed these values standardized values into the model and get an output, only I don't want the output in terms of being standardized, I want to be able to interpret it as it is 'naturally', I assume I multiply it by the standard deviation and then add the mean to it.

I am unsure if this method is correct, can anyone offer some assistance with how we introduce new test values into standardized data?

$\endgroup$

1 Answer 1

4
$\begingroup$

Imagine I have a large dataset of many variables and many observations. I would like to create a regression model to predict the values of new data. For the sake of ease, say I find a ridge regression model with some value 𝜆 serves my needs best. For my model, I decided to center and scale (standardize) my data before fitting the model. I then divide my dataset into training values and testing values, the model is built on the training values. I then test my model using the testing values and I am satisfied with the outcome.

Technically, there is leakage here. The best approach would be to split, then standardize. Everything you do the raw data is part of The Model (I wrire more about the distinction between a model and The Model here).

I should 'standardize' this observation, correct?

Yes, new predictions should be standardized using the mean and standard deviation from the training data.

$\endgroup$
2
  • $\begingroup$ If I may, you're saying it is best to divide the data into training and testing data, standardize the training data and then 'standardize' the testing data using the appropriate mean and standard deviation from the training data? And then after training and testing is done, any new test values should be standardized using the training datas values? $\endgroup$
    – TerryStone
    Apr 30, 2023 at 15:16
  • $\begingroup$ @TerryStone yes, the $x' = (x - \bar{x})/s$ bit that you are wanting to do, you should consider that as part of your model. So, $\bar{x}$ and $s$ are parameters that you want to learn as part of your model, and the best place to learn them is from the training data if possible. $\endgroup$
    – Him
    Apr 30, 2023 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.