I have 1000+ records of dataset. After I clean all the training data then I perform the GridSearchCV library to the training data. So the cleaning step that I've done is filling some missing value and I use mean of entire training data. But in GridSearchCV it splitting the training set and test set to perform k-fold cross validation. So this meant GridSearhCV make the test data leakage? Because I think that I fill the mean by using entire training data. But in practical I think we should not use training data to fill the missing value of testing data.
1 Answer
Yes.
If you fill in missing values of some feature with the mean of the non-missing values on a data set, then subsequently using that data set to construct a cross validation estimate of test error suffers a problem from leakage. The proper procedure is to impute differently for each CV fold, using only the training data for that fold.
I'm not sure if this can be done with a convenience method like GridSearchCV
. I tend to avoid those tools, since they are almost by definition less flexible than writing your own code.
-
$\begingroup$ So if i want to write my own code with this statregy. Should I use the simple k-fold cross validation with training set and test set. Then switch the training set and test set to fill the NaN again and do k-fold. Right? $\endgroup$ Commented Jul 10, 2018 at 14:27