2
$\begingroup$

I have 1000+ records of dataset. After I clean all the training data then I perform the GridSearchCV library to the training data. So the cleaning step that I've done is filling some missing value and I use mean of entire training data. But in GridSearchCV it splitting the training set and test set to perform k-fold cross validation. So this meant GridSearhCV make the test data leakage? Because I think that I fill the mean by using entire training data. But in practical I think we should not use training data to fill the missing value of testing data.

$\endgroup$
3
$\begingroup$

Yes.

If you fill in missing values of some feature with the mean of the non-missing values on a data set, then subsequently using that data set to construct a cross validation estimate of test error suffers a problem from leakage. The proper procedure is to impute differently for each CV fold, using only the training data for that fold.

I'm not sure if this can be done with a convenience method like GridSearchCV. I tend to avoid those tools, since they are almost by definition less flexible than writing your own code.

$\endgroup$
  • $\begingroup$ So if i want to write my own code with this statregy. Should I use the simple k-fold cross validation with training set and test set. Then switch the training set and test set to fill the NaN again and do k-fold. Right? $\endgroup$ – Puntawat Ponglertnapakorn Jul 10 '18 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.