I'm taking a course on GLMs after a few years of using machine learning models. The good about GLMs is how the probabilistic model ties in with the estimation and evaluation. So I'm trying to transfer some of those to what I already know about ML tasks.

In XGBoost, some of the objective functions seem to coincide with the likelihood functions used in GLM estimation (poisson regression, logistic regression, gamma regression the ones I have in mind).

This is the version of the exponential family we are using in my course:

$$ f_Y(y; \theta, \phi) = exp\{\phi[y\theta - b(\theta)] \: + \: c(y;\phi)\} $$

One thing I've noticed about the deviance (metric and residuals) is that they only depend on the exponential family parameters and on the saturated model. They don't depend on how I estimated the location or canonical parameter. This is the deviance metric:

$$ D^*(y, \hat\mu) = \phi D(y, \hat\mu)= 2\{L(y, y) -L(y, \hat \mu)\} $$ Where $L(y, y)$ is the loglikelyhood for the saturated model. For the exponential family, you can re-write the deviance as a sum of the individuals sample, something called the deviance component:

$$ D(y, \hat\mu) = 2\sum^n_{i=1}d^2(y_i,\hat\mu_i) $$

And writing it as a function of the exponential family form:

$$ d^2(y_i,\hat\mu_i) = y_i(\tilde \theta_i - \hat \theta_i) + (b(\hat \theta_i) - b(\tilde \theta_i)) $$

From there it is not hard to build the deviance residuals.

I can derive all that without knowing how we got $\hat \theta_i$. As far as I can tell, this could be estimated by a boosted tree.

So in theory, what is stopping me from calculating the deviance metric, deviance residuals and inheriting their probabilistic properties (giving a known scale parameter) and the possibility of using residuals diagnostics to judge model fit? If it is possible, why isn't it common?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.