2
$\begingroup$

I have a dataset with 12 predictor variables and a binary response variable. There's 5960 observations. One of the predictor variables has 1,260 missing values so I'm using k-nearest neighbours to impute them. The distance metric is Euclidean and k = 1 is the default of the algorithm.

Question: Should I include the response variable in the k-nearest neighbours algorithm?

My thoughts were that this may introduce some sort of overfitting.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

No, you shouldn't use the response variable. If the neighbours of a sample end up being other samples that are close in target variable, you'll fill in your features by looking at those samples and your overall success will be more optimistic.

Take KNNImputer from sklearn, it takes the target variable as input (for conforming with the fit/transform pattern), but never uses it.

Improve this answer
$\endgroup$
2
  • $\begingroup$ What if this imputing is done using it's own training set and no test information is used in the imputation. Additionally leave-one-out technique can be applied by not including the observation in question 's response variable? @gunes $\endgroup$
    – Nikola
    Jul 30, 2022 at 15:15
  • $\begingroup$ The test set shouldn't be used in any case. A hold-out set while doing the imputation is also preferred, because the validation set is the de-facto test set for model selection (which also includes imputation by concept). KNN imputation is done using the features. Including the response variable as a feature is unusual. The imputation might end up favoring nearby responses and neglecting others even when the other features are similar. $\endgroup$
    – gunes
    Jul 30, 2022 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.