1
$\begingroup$

Suppose I know some true distribution $S(x)$, and I have a method of approximating $S$ based on a transformation of another distribution $G(x|\theta)$. We denote the approximation as $S^*(x|\theta)$. The context is from here.

From a few sources [e.g.], we know that minimizing the KL-Divergence between $S(x)$ and $S^*(x|\theta)$ gives the maximum likelihood estimation (MLE) of $\theta$. However, now I'm interested in estimating the uncertainty in $\theta$. In my case, $G(x|\theta)$ is a gamma PDF and so $\theta$ has 2 parameters. So the question is:

Is there any way to characterize the joint likelihood distribution of $\theta$ based on $S(x)$ and $S^*(x|\theta)$ alone?

Let's assume we have no prior on $\theta$, unless that makes it harder ...

I'm imagining sampling various values of $\theta$ and computing the KL-Divergence, but I'm not sure what relationship the distribution $D_{KL}\big(S(x) ~||~ S^*(x|\theta)\big)$ over $\theta$ has with the likelihood distribution of $\theta$.

Improve this question
$\endgroup$
1
  • $\begingroup$ Actually, I may have figured it out. Since the likelihood of $\theta$ is defined as $E_S[log(S^*(x|\theta))] = E_S[log(S(x))] - D_{KL}(S(x)~||~S^*(x|\theta))$, to define the likelihood based on the $D_{KL}$, we can simply add back the term $E_S[log(S(x))]$ ( a constant w.r.t. $\theta$), from which we can estimate the uncertainty, etc... I think...? $\endgroup$
    – jessexknight
    Commented Mar 22, 2021 at 13:28

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.