Both answers to this question on imputing missing values note that, when imputing missing values in a test set for model evaluation, the replacement values should be the ones calculated and used in the training process (not calculated anew on the test data).
The author of Hands-On Machine Learning with Scikit-Learn & TensorFlow also suggests the same:
...you should compute the median value on the training set, and use it to fill the missing values in the training set, but also don't forget to save the median value that you have computed. You will need it later to replace missing values in the test set when you want to evaluate your system, and also once the system goes live to replace missing values in new data.
For instance, if missing values are being replaced with the median, the process for test set evaluation should be:
- Create the train/test split
- Calculate the median values for numerical variable(s) in the training set and save this value
- Train the model on the training set
- Use the median value(s) saved in step 2 to fill the missing value(s) in the test set
- Evaluate the model's performance on the test set
This strikes me as counter-intuitive. Wouldn't the goal be to replicate the entire process/pipeline (including imputation process, variable selection, outlier detection/removal, etc.) on the test set to avoid data leakage? It seems this would more closely approximate the process when applied to new data since the evaluation would be "blind" to the values from the training set.
In general, with a multistep modeling procedure, cross-validation must be applied to the entire sequence of modeling steps. In particular, samples must be "left out" before any selection or filtering steps are applied.
While the context for the the above is (1) feature selection and (2) cross-validation, wouldn't the same rationale apply to (1) imputing missing variables and (2) a single train/test split? If not, why is it bad practice to re-compute imputed values on a training set?