1
$\begingroup$

I'm using XGBoost (python) to solve a regression problem. I instantiate the model like so:

import xgboost
mod = xgboost.XGBRFRegressor()

I would like to extract the number of model degrees of freedom from this, in order to compare performance with simpler models. I am aware that there are alternate approaches, such as cross validation, but I'm curious as to whether an xgboost model has a clearly defined DoF, and if that's possible to obtain. I've looked at the docs but cannot find any mention of model DoF.

$\endgroup$
2
$\begingroup$

XGBoost does not have "a clearly defined DoF" unfortunately.

Degrees of freedom are not well-defined for regression trees. This is even more pronounces in the case of gradient boosters where we use potentially hunderds of trees. Adding in the fact that we also regularise our fitting procedures both within a tree and across trees makes the whole concept even muddier. (So we cannot even see it is something like num_of_trees times mean_num_of_nodes_per_tree.)

There is some work on data pertubation methods that have been used for regression trees. Those can lead to concepts like Generalised DoF (e.g. Ye 1998 On Measuring and Correcting the Effects of Data Mining and Model Selection) but that these methods themselves are covariance-based approximations not particular to gradient booster (and let alone XGBoost in particular)). Your initial idea about using cross-validation for comparing model performance is the way to go. :)

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.