There seems to be a large body of applied research where distribution q is picked to minimize KL(q,p) where p is empirical distribution. Are there theoretical reasons to prefer this estimator? For instance, a theoretical reason to prefer MLE for estimating mean of a one-dimensional Gaussian with mean between -1 and 1 because it's admissible, and also as sample size goes to infinity, it minimizes maximum risk and Bayes risk.

  • 1
    $\begingroup$ Is the KL minimization you refer to is done while containing p? (Otherwise, the minimizer will be trivially q). If so- KL minimization can be seen as an orthogonal projection, under KL norm. $\endgroup$
    – JohnRos
    Commented Oct 27, 2011 at 22:34
  • 2
    $\begingroup$ @JohnRos - I think your comment is a good start for an answer here. Do you want to repost as an answer and perhaps expand on the meaning/significance of "orthogonal projection under KL norm"? Yaroslav, I notice you are still an active user - could you indicate if this goes someway to answer your question or not? Thanks $\endgroup$
    – Corvus
    Commented Feb 17, 2013 at 20:45
  • 1
    $\begingroup$ This needs more background or context, what is $q$? If $q \in Q$, where $Q$ is a family of distributions indexed by $\theta$ you can show some connection to maximum likelihood. If $Q$ can be any imaginable density, we will just wind up with the EDF every single time which isn't interesting. $\endgroup$
    – AdamO
    Commented Nov 17, 2023 at 17:58


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.