3
$\begingroup$

I stumbled upon the following quantity and I'm wondering if anyone knows of anywhere it has appeared in the stats literature previously. Here's the setting:

Suppose you will observe data $y\in \mathcal{Y}$ which is distributed according to the density $p(y\,\vert\,\theta)$ for some parameter $\theta$ having a prior density $p(\theta).$ The marginal likelihood of $y$ is $p(y) = \int p(y\,\vert\,\theta)\,p(\theta)\,\text{d}\theta.$ Let $\theta_\text{ml}(y)$ be the maximum likelihood estimator of $\theta$ based on $y.$

The quantity of interest is $$c=\sup_{y\in \mathcal{Y}} \frac{p(y\,\vert\,\theta_\text{ml}(y))}{p(y)}.$$

I know that $c > 1,$ but I don't know much more than that. It certainly appears impossible to compute except perhaps in the simplest settings. I can't say immediately whether or when it is finite. I'm sure that it's also hard to bound above, which is what I'd be most interested in doing.

$\endgroup$
2
  • $\begingroup$ What is $m(y)$ in the denominator? Did you mean $p(y)$? $\endgroup$ Feb 13, 2018 at 7:46
  • $\begingroup$ Yes, sorry. Fixed it. $\endgroup$
    – stats_qs
    Feb 13, 2018 at 14:00

1 Answer 1

3
$\begingroup$

The quantity $p(y|\theta_\text{ml}(y))$ is the marginal likelihood for the most favourable prior, which is not a true prior since it depends on $y$. It simply achieves the upper bound. Taking the supremum over all $y$'s does not make sense from a Bayesian perspective, since it should be conditional on $y$.

Now if one considers the Normal-Normal case, with prior $\theta\sim\mathcal{N}(0,1)$ and observation $X\sim\mathcal{N}(\theta,1)$, we have $$p(y|\theta_\text{ml}(y))=\frac{1}{\sqrt{2\pi}}\qquad \text{and}\qquad m(y)=\frac{1}{\sqrt{4\pi}}\exp\{-y^2/4\}$$ Hence $$\sup_{y\in \mathcal{Y}} \frac{p(y\,\vert\,\theta_\text{ml}(y))}{m(y)}=\infty$$ which is not very surprising in that $p(y|\theta_\text{ml}(y))$ will correspond to an overfit of the model, while $m(y)$ includes a penalty à la Schwarz (BIC). This may also connects with Lindley's paradox.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.