Chapter 3 of Tom M. Mitchell. Machine Learning (free) says:

the available data has been split into three subsets: the training examples, the validation examples used for pruning the tree, and a set of test examples used to provide an unbiased estimate of accuracy over future unseen examples.

I understand this conceptually and am trying to understand this operationally.

Could someone please give an concrete example to illustrate how exactly a validation set being used for pruning a decision tree?

for example, how a exactly validation set in iris dataset for pruning a decision tree?

Note: this post is different to the one discussing how to divide dataset. this post focus on pruning.


1 Answer 1


The decision tree is applied to the validation data and the tree's parts are evaluated - and if necessary pruned - based on the chosen pruning strategy.
The simplest example of this that comes to mind is reduced error pruning: starting from each leaf node, the next higher split node is simply replaced by the majority class occuring there and if the prediction quality on the validation data is unchanged (or within a defined margin) then this change is kept. This scheme continues until all old as well as new leaf nodes have been examined.

As you can see, the validation set is used to assess how well the tree generalizes to data outside the training set - maybe a certain split node made sense for the training data, but actually ended up making the tree worse on other data. Pruning is used here to reduce the complexity of the tree and make it more general in an attempt to reduce overfitting.

  • $\begingroup$ thanks for your answer. would you please name a concrete dataset, such as iris, regarding which what kind of node learned by ID3 is to be pruned. $\endgroup$
    – JJJohn
    Dec 5, 2019 at 9:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.