3
$\begingroup$

I am trying to use the Kullback-Leibler as an R2 value for goodnes-of-fit for GLM models.

The R package performance defines their function as:

klr2 <- 1 - (model$deviance / model$null.deviance)

where deviance = 2 * loglik(full model) - loglik(reduced or fitted model) and

null deviance = 2 * loglik(full model) - loglik(null model),

based on An R-squared measure of goodness of fit for some common nonlinear regression models, which describes the Kullback-Leibler as an R2 value as the "proportionate reduction in uncertainty due to inclusion of regressors".

Given this calculation, it seems like the klr2 should be low if the deviance numerator has a high log likelihood for the reduced model. However, the literature paints the klr2 as a higher-is-better value, because we have greatly (if high) reduced the uncertainty in the model by adding our regressors.

Is a better goodness-of-fit with the Kullback-Leibler as an R2 value a higher or lower value? In particular, I am comparing the klr2 value of a full model to a reduced model, since I do not think I can compare the klr2 across models with different response variables.

$\endgroup$
4
  • 1
    $\begingroup$ This is in the wrong place but it seems to be so obvious that I'm wondering why it even needs to be asked. Ask yourself the results of the calculation if the deviance does not change, i.e. the model is not any good.That should give you the answer you need. $\endgroup$
    – DWin
    Oct 3, 2023 at 1:02
  • $\begingroup$ @IRTFM Should it be on stackexchange? And do you mean if the model deviance does not change? Do you mean if the log likelihood of the full model is the same as the log likelihood of the fitted model? I'm just trying to learn here. $\endgroup$
    – simpson
    Oct 4, 2023 at 3:33
  • 1
    $\begingroup$ SO is not the right place to post statistics questions even if you are “just trying to learn.” $\endgroup$
    – DWin
    Oct 4, 2023 at 4:33
  • $\begingroup$ @IRTFM Can you tell me where I should post this question to get an answer? $\endgroup$
    – simpson
    Oct 5, 2023 at 5:12

1 Answer 1

2
$\begingroup$

Just by calculus, it is seen that the model$deviance and the klr2 move in opposite directions. Therefore, if the klr2 is a "higher-is-better" value, then it must be the case that model$deviance is a "lower-is-better" value, so let's look into why.

The model deviance $(D)$ is related to the log-likelihood $(LL)$ of the full model and the fitted model: $$ D_{\text{model}} = 2\left( LL(M_{\text{full}}) - LL(M_{\text{fitted}}) \right) $$

More likely models are better (that's why we maximize likelihood instead of minimize likelihood), so the higher the $LL(M_{\text{fitted}})$ term, the better. $LL(M_{\text{full}})$ is a property of the data and is essentially a constant, so we can write something like $D_{\text{model}} \sim - LL(M_{\text{fitted}})$. Since $LL(M_{\text{fitted}})$ is better when it is higher, $-LL(M_{\text{fitted}})$ is better when it is lower. Thus, $D_{\text{model}}$ is a "lower-is-better" value, so klr2 is a "higher-is-better" value.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.