I'm having trouble understanding identifiability. Specifically, I'm not sure, in the following example, why $P\left(C\right)$ cannot be identified. Here's the example:

You have 2 unfair coins with the following probabilities:

$$P\left(H|C_1\right)=1/4$$ $$P\left(H|C_2\right)=3/4$$

Each coin has an equal probability of being chosen, so $P\left(C=C_1\right)=P\left(C=C_2\right)=1/2$.

My question is, given that I only observe $H$, why can't I identify $P\left(C\right)$ using basic algebra. That is, \begin{eqnarray} P\left(H\right)&=&P(C_1)P\left(H|C_1\right) + P\left(C_2\right)P\left(H|C_2\right)\\ &=& P\left(C\right)\left(P\left(H|C_1\right)+P\left(H|C_2\right)\right)\\ P\left(C\right) &=& \frac{P\left(H\right)}{P\left(H|C_1\right)+P\left(H|C_2\right)} \end{eqnarray}

In this case $P\left(C\right)$ would seem to be identifiable, so I'm not sure where I'm going wrong. Note that I'm not looking to identify $C$ itself (the actual coin), I want to know the distribution on $C$.

  • $\begingroup$ What does P(C) mean? Is it to represent the probability distribution for the chosen coin? In other words P(C=H)=1-P(C=T). If so how does P(C=H) relate to P(C=C1)? I think there is an algebra mistake at the point when you factor P(C) but I can't be sure because you didn't explain what it represents. $\endgroup$ – Michael R. Chernick Jul 12 '12 at 21:54
  • $\begingroup$ $P\left(C\right)$ represents the probability of picking either $C_1$ or $C_2$. $\endgroup$ – Phillip Cloud Jul 12 '12 at 22:23
  • $\begingroup$ How does one "observe" a probability?? Also, the probability of picking either $C_1$ or $C_2$ equals $1$ by the axioms. $\endgroup$ – whuber Jul 12 '12 at 22:27
  • $\begingroup$ @whuber $P\left(C\right)$ is the probability distribution over the set of possible coins. Is that correct? I'm honestly asking, I don't want you to think I'm being a smart-ass $\endgroup$ – Phillip Cloud Jul 12 '12 at 23:29

Identifiability problems arise in parameter estimation when you have two or more different sets of parameter values that give rise to the same probabilities of observing the random variable. In that case, you can't distinguish between the different possible values of the parameters by looking at the data, no matter how much data you have. (Writing loosely here...)

In your case, I suspect you're slightly misunderstanding what you observe and what you aren't. Typically you would observe the outcome of the coin flip $H$ or $T$, and you wouldn't know the probabilities, because if you did identifiability is a moot issue (since you've no parameters to estimate.)

In this case, let us assume that you DON'T know $P(H|C_1)$, $P(H|C_2)$, and $P(C_1)$, and all you observe is some collection of heads and tails. I assume you are interested in estimating the three probabilities.

Now, note that if $P(H|C_1)=0.75$, $P(H|C_2)=0.25$, and $P(C_1)=0.25$, then:

$P(H) = P(H|C_1)P(C_1) + P(H|C_2)P(C_2) = 0.75*0.25 + 0.25*0.75 = 6/16$

But if $P(H|C_1)=0.25$, $P(H|C_2)=0.75$, and $P(C_1)=0.75$, then, skipping the detail, $P(H) = 6/16$ again. In fact, there are an infinite number of parameter values (the probabilities) that will generate $P(H)=6/16$, or any other value you care to pick. $P(C_1)$ can be any probability, and you can find a $P(H|C_1)$ and $P(H|C_2)$ that will generate any specific $P(H)$. Consequently, no sequence of heads and tails will allow you to estimate $P(C_1)$. $P(C_1)$ is not identifiable.

  • $\begingroup$ I think you formulated the problem that the OP was trying to explain and solved it illustrating the unidentifiability. A big +1. $\endgroup$ – Michael R. Chernick Jul 12 '12 at 23:30
  • $\begingroup$ @Michael Chernick Agreed. $\endgroup$ – Phillip Cloud Jul 12 '12 at 23:53
  • $\begingroup$ Can one then say that if the parameters are linearly dependent then they are unidentifiable? $\endgroup$ – Phillip Cloud Jul 13 '12 at 0:07
  • $\begingroup$ @MichaelChernick - thanks! cpcloud - yes, in that case they are not identifiable, but that's not the only case. Imagine $x \sim \text{Exponential}(\theta_1/\theta_2)$, $\theta_1$ and $\theta_2$ are not identifiable. $\endgroup$ – jbowman Jul 13 '12 at 0:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.