4
$\begingroup$

I encountered the following question in my research and I hope this is the correct place to post it. I'm following the notation in this lecture note by Michael I. Jordan.

Assume random vector $X$ follows some exponential family distribution:

$$p(X=x|\eta)=\exp(\eta^T T(x)-A(\eta))f(x),$$

where $\eta$ is the vector of natural parameters and $T(x)$ is the vector of sufficient statistics. As an exponential family, it can also be parametrized by the mean parameter vector:

$$\mu = \mathbb{E}_{p(x|\eta)}[T(X)] = \frac{\partial A}{\partial \eta^T}(\eta).$$

Let $\mathbf{x}_n = (x_1,...,x_n)$ be a data vector composed of i.i.d. samples from this distribution. I would like to do maximum likelihood estimation for the parameter $\eta$. It can be shown that the MLE of the mean parameter is:

$$\hat{\mu}_{ML} = \frac{1}{n}\sum_{i=1}^n T(x_i),$$

so mathematically we have:

$$\hat{\eta}_{ML} = \left(\frac{\partial A}{\partial \eta^T}\right)^{-1}(\hat{\mu}_{ML}).$$

However, in practice, usually $A(\eta)$ is only given by the normalization condition $\int p(x|\eta) dx = 1$ and it is not explicitly known. $A(\eta)$ itself is also hard to evaluate numerically because $X$ may be of high dimension. In view of this, I'm wondering in this case how should we get the value of $\hat{\eta}_{ML}$?

$\endgroup$
3
$\begingroup$

In most (standard) exponential families, the moment generating function $A)\eta)$ is available in closed form. Here is a screen-copy from Wikipedia that shows the beginning of a list of examples, each of which can be easily inverted:

enter image description here

$\endgroup$
3
  • $\begingroup$ Thank you for your answer. The distribution I encountered doesn't fall into those standard exponential family distributions. Actually here the $X$ is of very high dimension, and I'm pretty confident that the closed-form expression for $A(\eta)$ is not known. $\endgroup$ – user166061 Feb 28 at 20:02
  • $\begingroup$ I then suggest you include your problem in the question itself. $\endgroup$ – Xi'an Feb 28 at 20:10
  • $\begingroup$ It's a statistical mechanics problem and its details are not very related to this question. Also I think this question itself is also valid in a broader context. The only background information I want to borrow from the original problem are 1. $X$ is of high dimension and 2. the closed-form of $A(\eta)$ is not available. $\endgroup$ – user166061 Feb 28 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.