# Understanding basic identifiability

I'm having trouble understanding identifiability. Specifically, I'm not sure, in the following example, why $P\left(C\right)$ cannot be identified. Here's the example:

You have 2 unfair coins with the following probabilities:

$$P\left(H|C_1\right)=1/4$$ $$P\left(H|C_2\right)=3/4$$

Each coin has an equal probability of being chosen, so $P\left(C=C_1\right)=P\left(C=C_2\right)=1/2$.

My question is, given that I only observe $H$, why can't I identify $P\left(C\right)$ using basic algebra. That is, \begin{eqnarray} P\left(H\right)&=&P(C_1)P\left(H|C_1\right) + P\left(C_2\right)P\left(H|C_2\right)\\ &=& P\left(C\right)\left(P\left(H|C_1\right)+P\left(H|C_2\right)\right)\\ P\left(C\right) &=& \frac{P\left(H\right)}{P\left(H|C_1\right)+P\left(H|C_2\right)} \end{eqnarray}

In this case $P\left(C\right)$ would seem to be identifiable, so I'm not sure where I'm going wrong. Note that I'm not looking to identify $C$ itself (the actual coin), I want to know the distribution on $C$.

• What does P(C) mean? Is it to represent the probability distribution for the chosen coin? In other words P(C=H)=1-P(C=T). If so how does P(C=H) relate to P(C=C1)? I think there is an algebra mistake at the point when you factor P(C) but I can't be sure because you didn't explain what it represents. Commented Jul 12, 2012 at 21:54
• $P\left(C\right)$ represents the probability of picking either $C_1$ or $C_2$. Commented Jul 12, 2012 at 22:23
• How does one "observe" a probability?? Also, the probability of picking either $C_1$ or $C_2$ equals $1$ by the axioms.
– whuber
Commented Jul 12, 2012 at 22:27
• @whuber $P\left(C\right)$ is the probability distribution over the set of possible coins. Is that correct? I'm honestly asking, I don't want you to think I'm being a smart-ass Commented Jul 12, 2012 at 23:29

Identifiability problems arise in parameter estimation when you have two or more different sets of parameter values that give rise to the same probabilities of observing the random variable. In that case, you can't distinguish between the different possible values of the parameters by looking at the data, no matter how much data you have. (Writing loosely here...)

In your case, I suspect you're slightly misunderstanding what you observe and what you aren't. Typically you would observe the outcome of the coin flip $H$ or $T$, and you wouldn't know the probabilities, because if you did identifiability is a moot issue (since you've no parameters to estimate.)

In this case, let us assume that you DON'T know $P(H|C_1)$, $P(H|C_2)$, and $P(C_1)$, and all you observe is some collection of heads and tails. I assume you are interested in estimating the three probabilities.

Now, note that if $P(H|C_1)=0.75$, $P(H|C_2)=0.25$, and $P(C_1)=0.25$, then:

$P(H) = P(H|C_1)P(C_1) + P(H|C_2)P(C_2) = 0.75*0.25 + 0.25*0.75 = 6/16$

But if $P(H|C_1)=0.25$, $P(H|C_2)=0.75$, and $P(C_1)=0.75$, then, skipping the detail, $P(H) = 6/16$ again. In fact, there are an infinite number of parameter values (the probabilities) that will generate $P(H)=6/16$, or any other value you care to pick. $P(C_1)$ can be any probability, and you can find a $P(H|C_1)$ and $P(H|C_2)$ that will generate any specific $P(H)$. Consequently, no sequence of heads and tails will allow you to estimate $P(C_1)$. $P(C_1)$ is not identifiable.

• I think you formulated the problem that the OP was trying to explain and solved it illustrating the unidentifiability. A big +1. Commented Jul 12, 2012 at 23:30
• @Michael Chernick Agreed. Commented Jul 12, 2012 at 23:53
• Can one then say that if the parameters are linearly dependent then they are unidentifiable? Commented Jul 13, 2012 at 0:07
• @MichaelChernick - thanks! cpcloud - yes, in that case they are not identifiable, but that's not the only case. Imagine $x \sim \text{Exponential}(\theta_1/\theta_2)$, $\theta_1$ and $\theta_2$ are not identifiable. Commented Jul 13, 2012 at 0:24

To add to @jbowman great answer, I believe this identifiability problem can be regarded also as a $$\textit{Label Switching}$$ problem.

The likelihood can be written as

$$p_{1}f_{C=1}(H)+p_{2}f_{C=2}(H) = \sum_{k=1}^{2}p_{k}f_{C=k}(H)$$

where $$p_{1}+p_{2}= P(C=C_{1})+P(C=C_{2})=1$$ and $$f_{C=i}(H)=P(H|C=C_{i})$$ with $$i=1,2$$.

And the problem of identifiability arises, because the likelihood is not invariant under permutations, i.e. we can change the labels of the components and acquire a likelihood that is equivalent to the original

$$p_{2}f_{C=2}(H) + p_{1}f_{C=1}(H)=\sum_{k=1}^{2}p_{k}f_{C=k}(H)$$