I have read many times that the AIC serves as an estimate of the KL divergence, and I know that AIC can be a negative value (and have seen that myself). Yet, the KL divergence must always be positive. I've not been able to find a simple formula for translating AIC into KL divergence, so what exactly is the relationship here? How is AIC estimating KL divergence?

  • 1
    $\begingroup$ It’s not an estimate of the KL divergence, it is an estimate of $a \mbox{KL} + b$ where $a$ and $b$ don’t depend on the models being compared. Hence minimizing AIC approximates minimizing the KL divergence between the model and truth. $\endgroup$ – guy May 3 '18 at 3:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.