4
$\begingroup$

How to use log partition function to derive $\mathbb{E}(X^2+Y^2)$, where $(X,Y)$ is from standard bivariate normal distribution? By standard bivariate normal I mean $\mu_x=\mu_y=0$ and $\sigma^2_X=\sigma^2_Y=1$ and $\sigma_{XY}=\rho$. The answer is clearly $\mathbb{E}(X^2+Y^2)=2$ but I don't know how to use the log partition function to calculate it. My calculation gave me $\mathbb{E}(X^2+Y^2)=-2\rho^2$.

Here is what I've done.

The p.d.f of $(X,Y)$ is $$f_{X,Y}(x,y)=\frac{1}{2\pi\sqrt{1-\rho^2}}\exp\left\{-\frac{1}{2(1-\rho^2)}\left(x^2+y^2-2\rho xy\right)\right\}$$ Thus we can write $f_{X,Y}(x,y)$ in a exponential family form:

$$f_{X,Y}(x,y)=\exp\left\{(x^2+y^2,xy)\cdot(-\frac{1}{2(1-\rho^2)},\frac{\rho}{1-\rho^2})-\frac{1}{2}\log{(1-\rho^2)}\right\},$$ from where we can see that the sufficient statistic $$T=(X^2+Y^2, XY)'$$ and the natural parameter $$\eta = (\eta_1,\eta_2)'= \left(-\frac{1}{2(1-\rho^2)}, \frac{\rho}{1-\rho^2}\right)'$$ and the log-partition function is $$A(\eta)=\frac{1}{2}\log{(1-\rho^2)}=\frac{1}{2}\log{(1-\frac{\eta_2^2}{4\eta_1^2})}$$

Since we can derive the expected value of $T$ using $$\mathbb{E}(T)=\nabla_\eta A(\eta)$$ then in our case we have $$\nabla_\eta A(\eta)=\left(\frac{\eta_2^2}{4\eta_1^3-\eta_2^2\eta_1}, \frac{\eta_2}{\eta_2^2-4\eta_1^2}\right)'$$

I am pretty confidence about the above derivation as I used an online tool to calculate it.

However, if I plug $\eta_1= -\frac{1}{2(1-\rho^2)}$ and $\eta_2= \frac{\rho}{1-\rho^2}$, what I got is $$\nabla_\eta A(\eta)|_{\eta=\eta(\rho)}=\left(-2\rho^2,-\rho\right)'=\mathbb{E}(T_1,T_2)=\mathbb{E}(X^2+Y^2,XY)$$

But this is apparently not correct as we all know that $\mathbb{E}(XY)=\rho$ and $\mathbb{E}(X^2+Y^2)=2$.

Where I did wrong?

$\endgroup$
2
$\begingroup$

This is a curved exponential family, which is a subset of the bigger full-rank family $$ \text{density} \propto \exp\big( \eta_1(x^2 + y^2) + \eta_2 xy - A(\eta)\big). $$ Consider computing the log-partition function for this larger family. You can write down the density for $(X,Y)$ assuming that $\text{var}(X) = \text{var}(Y) = \sigma^2$ and $\text{cor}(X,Y) = \rho$, that is, correlation coefficient $ = \rho$. By comparing the density to the above form you can figure out $\eta_1$ and $\eta_2$ and $A(\eta)$.

What you are trying to do is to recover $A(\eta)$ from only knowing $A(\tilde\eta(\rho))$ for some function $\widetilde \eta: \mathbb R \to \mathbb R^2$. This in general is not doable. There are many functions $A: \mathbb R^2 \to \mathbb R$ such that $A(\widetilde \eta(\rho)) = \frac12 \log(1-\rho^2)$.

Don't read the rest and try it out yourself.


The correct log-partition function for the full family turns out to be something like this: $$ A(\eta) = -\frac12 \log \Big( \frac14 \big(\eta_1^2 - \frac14 \eta_2^2\big) \Big) $$ with $\sigma^2 = -(\eta_1/2) /(\eta_1^2 - \eta_2^2/4)$.

$\endgroup$
3
  • $\begingroup$ Thank you. I see your point. Is there any way of not by introducing a new parameter, e.g. $\sigma^2$? Is there any general way of getting the correct log-partition function for the curved exponential family? $\endgroup$
    – Tan
    Jan 19 at 0:37
  • 1
    $\begingroup$ You can try to integrate the expression for the full-rank model. BTW, there is no unique curved family. You can get different curved families from the full-rank model. Pick any reasonable function $g : \mathbb R \to \mathbb R^2$ and use the reparametrization, $\eta = g(t)$. The log-partition function of the resulting model does not have enough information (as far as I understand) to reconstruct the log-partition function of the original. $\endgroup$
    – passerby51
    Jan 19 at 0:47
  • $\begingroup$ Exactly. I agree with you that different $g$ can give different log-partition function. However, integrating the expression for the full-rank model, e.g. $A(\eta) = \int e^{T'\eta}\mu(dx),\eta\in\Xi$ is not easy sometimes. Maybe a clever way is to notice the corresponding full-rank family with parameter $\theta$, where $\eta = \eta(\theta)$, e.g. noticing $\sigma^2$ in this case. $\endgroup$
    – Tan
    Jan 19 at 0:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.