5
$\begingroup$

Let $x$ and $y$ be random variables whose probabilities depend on an unknown parameter $\theta$. I am specifically interested in the case that both $x$ and $y$ are Bernoulli, but the question below can also be asked in general. Assuming sufficient regularity, the Fisher information of a set $X$ of random variables about $\theta$ is given by the expression:

$ \mathcal{I}_X(\theta)= -E\left[\left.\frac{\partial^2}{\partial\theta^2}\log f(X;\theta)\,\right|\theta\right]\enspace , $

where $f(X;\theta)$ is the probability density of $X$ conditional on $\theta$.

If $x$ and $y$ are independent, then:

$\mathcal{I}_{x,y}(\theta)= \mathcal{I}_{x}(\theta)+\mathcal{I}_{y}(\theta)\enspace .$

Question: Is it always true (even when $x$ and $y$ are dependent) that $\mathcal{I}_{x,y}(\theta)\leq \mathcal{I}_{x}(\theta)+\mathcal{I}_{y}(\theta)$?

My intuition from Shannon information tells me that surely this is so- if you have a coin, and you toss it once, your optimal strategy for the second toss is surely just to toss it again. But I've been unable to find a proof for this, and the analysis gets very complicated very fast when I try to prove this "subadditivity" property analytically.

Question: Do you have a reference for this property?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You should take a look at Stam's proof which is the original proof of Fisher information sub-addivity, and maybe on this one too which uses a generalized result. This paper contains a proof of a related result. Hope it helps!

-- edit You're right, this is another property, sorry for being misleading. According to my PhD advisor, it does not hold; sometimes super-additivity, (other inequalities).

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks! Is this the correct subadditivity property though? It looks to me like subadditivity with respect to something else- namely $\mathcal{I}_{T(x,y)}\leq \mathcal{I}_{x,y}$ for any function T, where x,y are iid (equation 1.5 in Stam). Have I missed something? $\endgroup$
    – Daniel Moskovich
    Commented Jun 11, 2015 at 21:42
  • $\begingroup$ I've skimmed through all three references you gave now, and can't find the statement of subadditivity in the sense that I need it- could you point me to it? Note in particular that a Bernoulli random variable does not have a continuous probability density function (of course); the references seem to all be working with condituous pdfs. $\endgroup$
    – Daniel Moskovich
    Commented Jun 12, 2015 at 6:09
  • $\begingroup$ Thank you for this. I now see that the opposite sometimes holds (because the joint holds more information than the sum of the marginals)- but I am still pretty sure that we have subadditivity in the Bernoulli case. Thinking further... $\endgroup$
    – Daniel Moskovich
    Commented Jun 12, 2015 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.