5
$\begingroup$

Let $x$ and $y$ be random variables whose probabilities depend on an unknown parameter $\theta$. I am specifically interested in the case that both $x$ and $y$ are Bernoulli, but the question below can also be asked in general. Assuming sufficient regularity, the Fisher information of a set $X$ of random variables about $\theta$ is given by the expression:

$ \mathcal{I}_X(\theta)= -E\left[\left.\frac{\partial^2}{\partial\theta^2}\log f(X;\theta)\,\right|\theta\right]\enspace , $

where $f(X;\theta)$ is the probability density of $X$ conditional on $\theta$.

If $x$ and $y$ are independent, then:

$\mathcal{I}_{x,y}(\theta)= \mathcal{I}_{x}(\theta)+\mathcal{I}_{y}(\theta)\enspace .$

Question: Is it always true (even when $x$ and $y$ are dependent) that $\mathcal{I}_{x,y}(\theta)\leq \mathcal{I}_{x}(\theta)+\mathcal{I}_{y}(\theta)$?

My intuition from Shannon information tells me that surely this is so- if you have a coin, and you toss it once, your optimal strategy for the second toss is surely just to toss it again. But I've been unable to find a proof for this, and the analysis gets very complicated very fast when I try to prove this "subadditivity" property analytically.

Question: Do you have a reference for this property?
$\endgroup$
1
$\begingroup$

You should take a look at Stam's proof which is the original proof of Fisher information sub-addivity, and maybe on this one too which uses a generalized result. This paper contains a proof of a related result. Hope it helps!

-- edit You're right, this is another property, sorry for being misleading. According to my PhD advisor, it does not hold; sometimes super-additivity, (other inequalities).

$\endgroup$

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

  • $\begingroup$ Thanks! Is this the correct subadditivity property though? It looks to me like subadditivity with respect to something else- namely $\mathcal{I}_{T(x,y)}\leq \mathcal{I}_{x,y}$ for any function T, where x,y are iid (equation 1.5 in Stam). Have I missed something? $\endgroup$ – Daniel Moskovich Jun 11 '15 at 21:42
  • $\begingroup$ I've skimmed through all three references you gave now, and can't find the statement of subadditivity in the sense that I need it- could you point me to it? Note in particular that a Bernoulli random variable does not have a continuous probability density function (of course); the references seem to all be working with condituous pdfs. $\endgroup$ – Daniel Moskovich Jun 12 '15 at 6:09
  • $\begingroup$ Thank you for this. I now see that the opposite sometimes holds (because the joint holds more information than the sum of the marginals)- but I am still pretty sure that we have subadditivity in the Bernoulli case. Thinking further... $\endgroup$ – Daniel Moskovich Jun 12 '15 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.