2
$\begingroup$

I have classification setup with $r$ classes and three random variables:

  • $T$: true category
  • $F^A$: category assigned by the (fallible) classifier "A"
  • $F^B$: category assigned by the (fallible) classifier "B"

For these, I have the following information:

  • $p_i := P(T = i) \quad (i \in \{1, \ldots, r\})$
  • $c^A_{ik} := P(F^A = k|T = i) \quad (i \in \{1, \ldots, r\}, \ k \in \{1, \ldots, r\})$
  • $c^{B|A}_{kj} := P(F^B = j|F^A = k) \quad (j \in \{1, \ldots, r\}, \ k \in \{1, \ldots, r\})$

As a result, I have

$$ \pi^A_k := P(F^A = k) = \sum_{i = 1}^{r}{P(T = i) \, P(F^A = k|T = i)} = \sum_{i = 1}^{r}{p_i \ c^A_{ik}} $$

and

$$ \pi^B_j := P(F^B = j) = \sum_{k = 1}^{r}{P(F^A = k) \, P(F^B = j|F^A = k)} = \sum_{k = 1}^{r}{\pi^A_k \ c^{B|A}_{kj}}\\ = \sum_{k = 1}^{r}{\sum_{i = 1}^{r}{p_i \ c^A_{ik}} \ c^{B|A}_{kj}} = \sum_{i = 1}^{r}{p_i \ \sum_{k = 1}^{r}{c^A_{ik}} \ c^{B|A}_{kj}}\\[30pt] $$

With vectors $p = (p_1, ..., p_r)^T$, $\pi^A = (\pi^A_1, ..., \pi^A_r)^T$, $\pi^B = (\pi^B_1, ..., \pi^B_r)^T$, and matrices $C^A = (c^A_{ik})_{1 \leq i, k \leq r}$ and $C^{B|A} = (c^{B|A}_{kj})_{1 \leq k, j \leq r}$, the above expressions are equivalent to

$$ (\pi^A)^T = p^T \, C^A \quad \text{and} \quad (\pi^B)^T = (\pi^A)^T \, C^{B|A} \, = p^T \, C^A \, C^{B|A} $$

So far, so good. Quite simple stuff. Now, I want to compute $c^B_{ij} := P(F^B = j|T = i)$. Using the law of total probability and the definition of conditional probabilities, I get

$$ P(F^B = j | T = i) = \sum_{k = 1}^{r}{P(F^B = j | F^A = k \wedge T = i) \, P(F^A = k|T = i)}. $$

From simulations (and intuition, because of the last matrix product of $(\pi^B)^T$), I concluded something slightly different:

$$ P(F^B = j | T = i) = \sum_{k = 1}^{r}{P(F^B = j | F^A = k) \, P(F^A = k|T = i)}. $$

The big question here: why is $P(F^B = j | F^A = k \wedge T = i) = P(F^B = j | F^A = k)$? The most simple answer would be that $T$ is independent of $F^A$ and $F^B$, but this is usually not true. What other reasons could there be? And how can I prove it? The sum expression of $\pi^B_j$ (and the matrix expression of $\pi^B$) is not sufficient (, right?).

Maybe you could provide me with links to some publications that already dealt with that problem.

$\endgroup$
1
$\begingroup$

The problem here is that your information is insufficient to solve the problem, since you have not specified the statistical relationship between $F^B$ and $T$, conditional on $F^A$. If you are willing to assume conditional independence then you have:

$$F^B \bot \text{ } T | F_A \quad \quad \iff \quad \quad \mathbb{P}(F^B=b |F^A=a) = \mathbb{P}(F^B=b|F^A=a, T=t)$$

In other words, making the probability equation you are using in your analysis is equivalent to assuming that the random variables $F^B$ and $T$ are conditionally independent given $F^A$. This means that once you already know $F^A$, your knowledge of the behaviour of $F^B$ is not affected by the true state $T$. If this is a reasonable assumption in your analysis then the use of that equation is acceptable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.