It's known that maximizing the log-likelihood is equivalent to minimizing the Kullback-Leibler divergence between the model $q(x \mid \theta)$ and the unknown true data generating distribution $p(x)$:

$$ \begin{split} \hat{\theta}_{\mathrm{ML}} &= \arg\min_{\theta} \mathrm{KL}(p(x) \mid\mid q(x \mid \theta)) = \arg\min_{\theta} \left[ \mathbb{E}_{X \sim p} \ln p(X) - \mathbb{E}_{X \sim p} \ln q(X \mid\theta) \right]\\ &= \arg\min_{\theta} \left[ \mathrm{const} - \mathbb{E}_{X \sim p} \ln q(X \mid\theta) \right]\\ &= \arg\max_{\theta} \mathbb{E}_{X \sim p} \ln q(X \mid\theta) \approx \arg\max_{\theta} \frac1N \sum_{n=1}^N \ln q(x_n \mid\theta) \end{split} $$

Is there anything similar for maximum a posteriori (MAP) estimation? EDIT: by "similar" I mean "something that involves a divergence between distributions (PDFs, CDFs, characteristic functions, etc), not values of $\theta$". Is MAP equivalent to minimization of some divergence between distributions that depend on $\theta$? Maybe some distributions of $\theta$, like the KL divergence for MLE as shown above?

Given the posterior:

$$ f(\theta \mid X) = \frac{q(X \mid \theta) r(\theta)}{\int_{\Theta} q(X \mid \theta) r(\theta) d\theta} $$

...does maximizing it $f(\theta \mid X) \to \max_{\theta}$ minimize some kind of divergence between some distributions, similar to how MLE minimizes the KL divergence?

I know the KL divergence between $f(\theta \mid X)$ and an approximation $f(\theta)$ is used in variational inference, but there we're looking for the entire approximate posterior $f(\theta)$, while MAP is for point estimates.

The usual reasoning for MAP seems to be "maximize the posterior to find its mode, which is the most probable/likely/common value of $\theta$", but I'm wondering whether MAP can be derived as a minimizer of some distance/divergence between distributions, not values of the parameters $\theta$.

EDIT: comments below point out that the median can be derived as a minimum distance solution. However, this distance is defined for the parameter space $\Theta$, while I'm looking for an interpretation that involves distances/divergences between probability distributions.

  • $\begingroup$ Nothing special about the posterior except in a MAP context. Your question is really what is minimised by the mode of a distribution. $\endgroup$
    – Henry
    Commented Jan 24, 2023 at 22:06
  • $\begingroup$ Suppose $m_k$ minimises $\mathbb E\left[|\Theta-m_k|^k\right]$ so $m_1$ is the median of the distribution for $\Theta$, and $m_2$ is the mean. Then consider $\lim\limits_{k \to 0^+} m_k$. In my view this is not a convincing cost measure, especially if the distribution is continuous $\endgroup$
    – Henry
    Commented Jan 24, 2023 at 22:10
  • $\begingroup$ @Henry, I'm wondering whether there's a different approach that doesn't directly involve the mode. Say, maximizing the likelihood is often motivated as looking for a $\theta$ which makes the data $\{x_n\}$ most "likely". The point of view of KL divergence doesn't have to mention this and involves minimizing information loss. You could generalize this to other divergences and distances and arrive at minimum distance estimation. So, we get a view of MLE which doesn't involve maximizing the likelihood. I'm asking if there's anything similar for MAP, something that doesn't involve modes. $\endgroup$
    – ForceBru
    Commented Jan 24, 2023 at 22:29
  • 1
    $\begingroup$ @ForceBru the answer of Henry does not involve modes directly. They're pointing out that there's a sense in which the mode minimizes $\ell_0$ loss; see here for more: stats.stackexchange.com/questions/329908/… $\endgroup$ Commented Jan 24, 2023 at 22:52
  • 1
    $\begingroup$ @ForceBru yeah im not sure, but one thing I would point out is that the MAP is only very tenuously a Bayesian point estimate. $\endgroup$ Commented Jan 24, 2023 at 23:10


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.