10
$\begingroup$

In the comments on this answer, user Scortchi asks:

So iff there's a sufficient statistic of constant dimension, there's a conjugate prior?

As far as I know this didn't get a complete answer, so I'm asking it as a new question in the hope of finding out whether it's true. My question is the quote above; I give more details below.

This question can be seen as asking for a generalisation of the Pitman-Koopman-Darmois theorem, which states that if a family of distributions is such that the support does not depend on the parameters, and if the family has a sufficient statistic whose dimensionality doesn't change as the number of samples increases, then the family must be an exponential family.

We also have that if a family of distributions is such that the support does not depend on the parameters and the family admits a conjugate prior, then it must be an exponential family, which is a similar but different result.

However, as the example in the linked answer shows, if we relax the assumption that the support doesn't depend on the parameters, then it's possible for a distribution to have a conjugate prior without being an exponential family. The question is whether something similar happens if we relax the corresponding assumption in the Pitman-Koopman-Darmois theorem, and specifically, whether we end up with the same set of families of distributions in both cases.

In other words, the conjecture is that the Pitman-Koopman-Darmois theorem can be generalised into the following statement: "For an arbitrary family of distributions, whose support might depend on its parameters, the family has a conjugate prior if and only if it has a sufficient statistic whose dimensionality doesn't change as the number of samples increases." Is this statement true or false?

$\endgroup$
0

1 Answer 1

9
$\begingroup$

If there exists a finite dimensional conjugate family, $$\mathfrak F=\{\pi(\cdot|\alpha)\,;\ \alpha\in A\}$$ with $\dim(A)=d$, this means that, for any $\alpha\in A$, there exists a mapping $\tilde\alpha_n\,:\mathfrak X^n \mapsto A$ such that $$\pi(\theta|x_{1:n},\alpha)\propto f_n(x_{1:n}|\theta)\pi(\theta|\alpha) \propto \pi(\theta|\tilde\alpha_n(x_{1:n}))$$ Hence, for an arbitrary $\alpha$, $$f_n(x_{1:n}|\theta) = \pi(\theta|\tilde\alpha_n(x_{1:n})) m_\alpha(x) \pi(\theta|\alpha)^{-1}$$ factorises into a function of $\theta$ and $\tilde\alpha_n(x_{1:n})$ and a function of $x$ that does not depend on $\theta$. This implies that $\tilde\alpha_n(x_{1:n})$ is a sufficient statistic when $\Theta$ is restricted to the support of $\pi(\cdot|\alpha)$. Furthermore, assuming $\alpha$ and $\alpha^\prime$ are such that the supports of $\pi(\cdot|\alpha)$ and of $\pi(\cdot|\alpha^\prime)$ intersect, each posterior is a function of both its summary statistic (e.g., attached with $\alpha$) and of the other (e.g., attached with $\alpha^\prime$), which means they must be functions of one another (i.e., in bijection), I believe, leading to a potential conclusion that is independent from the support.

Considering the converse implication, and assuming iid data, if there exists a sufficient statistic of fixed dimension for all (large enough) $n$'s, $t_n\,:\mathfrak X^n \mapsto \mathbb R^d$, then by the factorisation theorem, $$f(x_{1:n}|\theta) = \tilde {f_n}(t_n(x_{1:n})|\theta)\times m_n(x_{1:n})$$ which implies that, for any prior $\pi(\cdot)$, the posterior satisfies $$\pi(\theta|x_{1:n})\propto \tilde {f_n}(t_n(x_{1:n})|\theta)\times \pi(\theta)$$ For a given distribution density $\pi_0(\cdot)$ over $\Theta$, the family of priors $$\mathfrak F=\{ \tilde \pi(\theta)\propto \tilde {f_n}(t_n(x_{1:n})|\theta) \pi_0(\theta)\,;\ n\in \mathbb N, x_{1:n}\in\mathfrak X^n\}$$ where the $x_{1:n}$'s are pseudo-observations of size $n$ indexing the prior distributions, is conjugate since, if the prior writes $$\tilde \pi(\theta)\propto \tilde {f_n}(t_n(x_{1:n})|\theta) \pi_0(\theta)$$ the posterior associated with the observations $y_{1:m}$ writes $$\tilde \pi(\theta|y_{1:m})\propto \tilde {f_m}(t_m(y_{1:m})|\theta)\tilde {f_n}(t_n(x_{1:n})|\theta)\pi_0(\theta)\propto \tilde {f}_{n+m}(t_{n+m}(z_{1:(n+m)})|\theta)\pi_0(\theta) $$ where $z_{1:(n+m)}=(x_{1:n},y_{1:m})$.

$\endgroup$
4
  • 1
    $\begingroup$ Thank you, this is great. There is maybe a slightly tricky point in your third equation though: it seems like we have to assume that $\pi(\theta|\alpha)>0$ for every $\theta$ and the chosen value of $\alpha$. This isn't the case for any arbitrary $\alpha$ in the example I linked to, where the conjugate prior is the Pareto family. However, it seems like that might not be a problem, because you only really need there to exist some value of $\alpha$ for which it's the case, and that seems like it should always be true. Do you have any thoughts about that? $\endgroup$
    – N. Virgo
    Mar 14, 2021 at 4:19
  • $\begingroup$ You are correct, I missed that point. $\endgroup$
    – Xi'an
    Mar 14, 2021 at 10:13
  • $\begingroup$ The first implication would work were there a conjugate prior whose support included the support of the likelihood function. Or a finite collection of conjugate priors whose support union contained the support of the likelihood function. $\endgroup$
    – Xi'an
    Mar 14, 2021 at 10:29
  • $\begingroup$ I think there could be a possible resolution by arguing that the summary statistics $α_n(x_{1:n},α)$ are functions of one another when the prior hyperparameter α varies, but cannot quickly produce a rigorous argument, because of the varying supports (in θ and in $x_{1:n}$). $\endgroup$
    – Xi'an
    Mar 14, 2021 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.