0
$\begingroup$

I have a dataset and my intention would be to predict a binary variable using DecisionTreeClassifier.

dtc = DecisionTreeClassifier()
parameter_grid = {'criterion': ['gini', 'entropy'],
                  'splitter': ['best', 'random'],
                  'max_depth': range(1,25),
                  'min_samples_split': range(2,30),
                  'max_features': range(1,10)}

skf = StratifiedKFold(n_splits=15)
grid_search = GridSearchCV(dtc, param_grid=parameter_grid, scoring='recall', cv=skf)
grid_search.fit(X, Y)
print('Best score: {}'.format(grid_search.best_score_))
print('Best parameters: {}'.format(grid_search.best_params_))
#grid_search.grid_scores_

Using GridSearch I can find the best set of parameters of my model. The Score in output is the mean score on the test set? I am not understanding how GridSearch finds the best parameters using Kfold or StratifiedKfold. In this case X and Y represent all my database, with X predictors and Y target (0,1). So, when I run

grid_search.fit(X,Y)

is it correct to pass the entire dataset then using grid_search.best_estimator_ again on X or I should split before the database in train test, pass only train to the grid_search and then run grid_search.best_estimator_ on the test set?

$\endgroup$
1
$\begingroup$

I think it's important to step back and consider the purpose of breaking your data into a training and test set in the first place.

Ultimately, your goal is to build a model that will perform the best on a new set of data, given that it is trained on the data you have. One way to evaluate how well your model will perform on a new set of data is to break off some of your data into a "test" set, and only build your model on the remaining "training" set. Then, you can apply the model to your test set, and see how well it does in its prediction, with the belief that it will perform similarly to how it would perform on a new set of data.

Technically speaking, there's nothing wrong with doing grid search to tune hyperparameters on all of your data; you're free to build a model however you want. But by using grid search on all of your data, you are defeating the purpose of doing a training/test split. That's because if you do the training/test split after doing grid search on all of your data to tune hyperparameters, applying your model to the test set no longer gives an estimate of how well your model will perform on new data, since your model has seen the test set, in the sense that the data in the test set was used to tune the hyperparameters.

As a result, if you do grid search on all of your data, the error on your test set will be biased low, and when you go to apply your model to new data, the error could be much higher (and likely will, except for the effects of randomness).

In summary, you should only use gridsearch on the training data after doing the train/test split, if you want to use the performance of the model on the test set as a metric for how your model will perform when it really does see new data.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ thank you for your time. On the base of your comment, this procedure dezyre.com/recipes/… would not be correct, because gridsearch is applied on all dataset and then cross validation is applied on all dataset again to evaluate the performance of the model. Is it correct my comment? @Nick Koprowicz $\endgroup$ – zorro Mar 9 at 17:23
  • $\begingroup$ It looks like that example is just about how to tune the hyperparameters with grid search; they don't do any training/test split. So there's nothing wrong with that example. If all you want to do is find the best hyperparameters possible, then there's no need for a training/test split, and you can use all the data as they do in the example. But if you're doing a training/test split because you want to be able to estimate the error of your model on a new set of data by running your model on a test set, then you should only do the grid-search cv with the training data. $\endgroup$ – Nick Koprowicz Mar 9 at 19:12
0
$\begingroup$

GridSearch Cv will calculate the average of out of fold recall for each combination of parameters, the set of parameters with best score, will be chosen by Grid search CV. It is fine to use the entire dataset, as you are using Cv method, which will check the score on out of fold set, hence you are not evaluating performance on Training data (on which model is trained) for parameter selection. for example: we want to tune max depth of tree, let's say maximum depth parameter we want to test are 5,10,15. Grid Search Cv will calculate recall score on out of fold set for all three value. The max depth value corresponding to best score on out of fold set will be chosen. https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ ok thank you. But why here datascience.stackexchange.com/questions/51831/… the first answer said "You are correctly splitting your data into train/test, and then using your training data only to find optimal hyper-parameters". Because in my case instead I am passing all the dataset. $\endgroup$ – zorro Mar 9 at 10:35
  • $\begingroup$ Second question, is it correct using cross_val_score with the best estimator found and on all the data, always using CV, or I have to use grid.score ? $\endgroup$ – zorro Mar 9 at 10:40
  • $\begingroup$ @zorro, as you are using CV (so Grid search will use the left out set to decide the hyperparameter and that is what we want. We want to see results on dataset, which is not used in training. $\endgroup$ – Harshit Mehta Mar 12 at 3:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.