I always thought that there are only two ways the predictions from the final leaf are extracted in a tree based model:
- For regression problems take the average of the continuous variable to be predicted for the entire node
- For the classification problem make the mode of the label node as the final prediction
Then recently i started reading up on xgboost from the original paper by the authors of the algorithm and i realized i didnt quite understand how it is done? I have 2 reasons to believe my understanding is not right entirely. Here is the image for the explanation of CART from paper:
As is seen this is a classification problem (Does the person like the computer games?) and hence the output 2 for a node does not make sense to me
The paper also says $\lambda||w||^2$ is to be added as the regularization term in the cost function. $w$ is the weight of the terminal node (which i understand the final prediction in the node in a regression tree) in which case it does not make much sense to me to have this particular regularization term as it is just penalizing the high value of predictions and not making a model less or more complex
How are the predictions made in CART, what is the final score of a leaf, how is it obtained?
Here is an example.
Final leaves: 3 (classification):
first node: (1,1,1,0,0), second node: (1,1), third node: (1,1,1,1)
Final leaves: 3 (Regression):
first node: (0.1,0.2,0.3,100), second node: (4,4,4), third node: (50)
what would be the score and w (that is being referred to in regularization term in the paper) for each node in both cases?