I am looking at question 1b of the following notes: http://www.gatsby.ucl.ac.uk/teaching/courses/ml1/asst1.pdf

In 1a, I have shown that the Beta distribution has a density that can be written in the form $g(\boldsymbol{\theta}) f(\textbf{x}) e^{\phi(\boldsymbol{\theta})^T T(\textbf{x})}$ where:

$g(\boldsymbol{\theta}) = \frac{1}{B(\alpha,\beta)}$

$f(x) = \frac{1}{x(1-x)}$

$\phi(\boldsymbol{\theta}) = \begin{pmatrix} \alpha \\ \beta \end{pmatrix}$

$T(x) = \begin{pmatrix} \log{x} \\ \log{(1-x)} \end{pmatrix}$

Now, according to this thread, I can calculate the expectation of the sufficient statistic, $E[T(x)]$, by rewriting the density as

$f(\textbf{x}) e^{\phi(\boldsymbol{\theta})^T T(\textbf{x})} e^{-A(\boldsymbol{\theta})}$

and then using $E[T(x)]_i = \frac{A'(\boldsymbol{\theta})_i}{\phi'(\boldsymbol{\theta})_i}$

To do this, we define $A(\theta) = - \log{g(\theta)} = \log{B(\alpha,\beta)}$ but then I become uncertain of what to do. My attempt so far looks like:

$A'(\theta) = \frac{1}{B(\alpha,\beta)} \begin{pmatrix} B_\alpha \\ B_\beta \end{pmatrix}$ where $B_\alpha = \frac{\partial B}{\partial \alpha}$ etc. I also find $\phi'(\theta) = \begin{pmatrix} 1 \\ 1 \end{pmatrix}$.

I'm pretty sure I've made a mistake with the calculation of $phi'(\theta)$ since the vector of ones seems wrong, and yet $\phi'(\theta) = \frac{\partial}{\partial \theta_i} \phi(\theta) = \begin{pmatrix} \frac{\partial}{\partial \alpha} \\ \frac{\partial}{\partial \beta} \end{pmatrix} \phi(\alpha, \beta)$. Is this going to become a 2x2 matrix?

I have managed to get this technique to work for distributions such as binomial and Poisson where the sufficient statistic is a scalar. I am getting confused in this case where we have a vector of sufficient statistics.

  • 1
    $\begingroup$ Please make the question self-contained rather than expecting the reader to follow a web link. $\endgroup$ – Xi'an May 8 '19 at 8:14

The thread you provide only operates for one-dimensional exponential families. In general, when the exponential family is written wrt its natural parameter: $$f(x) e^{\theta^T T(x)} e^{-A(\boldsymbol{\theta})}$$ the mean of the associated (sufficient) statistic is $$\Bbb E_\boldsymbol\theta[T(X)] = \nabla A(\boldsymbol{\theta})$$ Since this is the case for the Beta distribution, namely $\phi( \boldsymbol\theta)=\boldsymbol\theta$, $$\nabla A(\boldsymbol{\theta})=\frac{1}{B(\alpha,\beta)} \begin{pmatrix} \frac{\partial B(\alpha,\beta)}{\partial\alpha} \\ \frac{\partial B(\alpha,\beta)}{\partial\beta} \end{pmatrix}$$

  • $\begingroup$ It seems that we cannot always have $\phi(\theta) =\theta$ however. For example, with the gamma distribution, I find $\phi(\theta) =( \alpha - 1, - \beta) ^T$. Under these circumstances, if I follow through the algebra, we would have $(\nabla \phi) E[T(x)] = \nabla A$. Is that correct? $\endgroup$ – user11128 May 26 '19 at 9:14
  • 1
    $\begingroup$ You can always reparameterise $\phi(\theta)$ as the default $\theta$. $\endgroup$ – Xi'an May 26 '19 at 10:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.