Decision tree model selection process

Trying to apply train, validation, and test set to a Binary Decision Tree classifier, following the logic of https://www.coursera.org/learn/machine-learning/home/welcome.

The logic is as follows, if you optimize parameters on the full training set and subsequently test performance on the test set. The parameters are optimized minimizing the error on the test set. This means the results are less generalizable since we fit the parameters using the training set, and testing results on the test set.

Example used in the course: Use linear regression to fit n-polynomial function to the data. How to decide on which degrees of n?

• split data into train (60), validation (20), test (20)
• use training set to build n polynomial functions and test results on the validation set.
• if we would test results on the test set the results become less generelizable.
• pick the n polynomial function that minimizes the validation error.
• use this n degree polynomial function to find optimal theta parameters using the training set.
• compute generalization error using the test set.

Currently, I split my data set into 60% training, 20% validation, and 20% test set.

I use the training set to perform a grid search to find optimal values for:

• min_samples_leaf
• min_samples_split
• criterion
• max_depth

and subsequently, GridSearch takes out the paramaters which maximizes the chosen performance measure. These parameters I use to build the final model to test the predictions.

I find it hard to apply the logic from the course, to the example of the decision tree. Can someone tell what I'm doing wrong, if so?

• What exactly is the problem? What you say you do sounds reasonable, I don't understand why or where you think things go wrong? – jonnor Apr 18 at 0:17