# How to calculate prior probabilities

I have two possible events $A$ and $B$ that could lead to $n$ possible consequences $X_1, X_2, \ldots , X_n$, $P(A) + P(B) = 1$, $P(X_1) + P(X_2) + \ldots + P(X_n) = 1$. I know all conditional probabilities $P(X_1 \mid A), P(X_1 \mid B), \ldots , P(X_n \mid A), P(X_n \mid B)$, and I need to find $P(A \mid X_1), P(B \mid X_1), \ldots , P(A \mid X_n), P(B \mid X_n)$.

According to the Bayes' rule,

$$P(A \mid X_1) = \frac{ P(A) P(X_1 \mid A)}{ P(A) P(X_1 \mid A) + P(B) P(X_1 \mid B)} .$$

The question is what is the proper way to calculate priors $P(A)$ and $P(B)$. Are they equal to each other? Are they equal to the proportions of their appearances in my dataset? In the latter case, should I calculate priors separately for each participant?

UPD. My data is basically the sequences of $X$'s for different participants.

• Probabilities are typically estimated using sample proportions if possible, and it generally wouldn't be reasonable to just assume $P(A) = P(B) = 1/2$. – dsaxton Jan 6 '16 at 0:31
• @dsaxton Thanks, initially for some weird reason I assumed that they were equal. I guess I should I set different priors for different participants since these proportions vary across them – Evgenii Nikitin Jan 6 '16 at 0:33
• Wait, I just realized that I know nothing about these proportions since I only have X's, not A and B for each observation. – Evgenii Nikitin Jan 6 '16 at 0:44
• In that case, with no information on the priors probabilities there's basically no way to arrive at the posterior ones, at least not without making assumptions about the priors. – dsaxton Jan 6 '16 at 0:48

Since you have the conditional probabilities (i.e. $P(X_i | A)$ and $P(X_i | B)$), and that $P(B) = 1 - P(A)$, you need to fix the value of only one variable, namely $P(A)$; let's denote it by $p$.
You have to pick $p$ that fits your data best. To do so, you can look at the log likelihood function and set $p$ so that the (log) likelihood function, denoted by $L(p)$, is maximized. In other words, you are looking for $p^*$ defined as: $$p^* = \arg\max_{0 \leq p \leq 1} L(p)$$ Let $D = \left\{ X^{(1)}, \dots, X^{(N)} \right\}$ be $N$ samples that you observed. Also, let $M$ be a $n \times 2$ matrix representing the conditional probabilities; i.e. $M[k, 1] = P(X_k | A)$ and $M[k, 2] = P(X_k | B)$. The log likelihood function on $D$ is defined as: \begin{align} L(p) &= \sum_{i=1}^N \log \left( P(A) P(X^{(i)} | A) + P(B) P(X^{(i)} | B) \right) \\ &= \sum_{i=1}^N \log \left( p \times M[X^{(i)}, 1] + (1 - p) \times M[X^{(i)}, 2] \right). \\ \frac{\partial L}{\partial p} &= \sum_{i=1}^n \frac{M[X^{(i)}, 1] - M[X^{(i)}, 2]}{p \times M[X^{(i)}, 1] + (1 - p) \times M[X^{(i)}, 2]} \end{align} Note that the log likelihood function is concave, because the second derivative is negative (this is easy to show). Assuming that $\nexists k, M[k, 1] = M[k, 2] = 0$, we have: \begin{align} p^* &= \begin{cases} 1 & \text{if } \frac{\partial L}{\partial p} > 0, \text{for } p \in (0, 1) \\ 0 & \text{otherwise} \end{cases} \end{align}
• What are $a$, $b$, and $D_i$? Note that $p^*$ turns out to be either 1 or 0, based on whether $\sum_{i=1}^N M[X^{(i)}, 1] > \sum_{i=1}^N M[X^{(i)}, 2]$ or not. This is assuming that you do not have any other information about the value of $p$. – Sobi Jan 6 '16 at 1:11
• @Evgenii Nikitin, please note that I updated my response. I had forgotten the $\log$ in the old one. Back to your question, I assumed that the matrix $M$ (i.e. the conditional probabilities) are fixed. If you have to tune those too then the problem becomes different; still very similar to what we did here though! You can always compute the gradient with respect to each parameter in the model and move in the direction of the gradient vector (i.e. perform gradient descent) to maximize the log likelihood function. – Sobi Jan 6 '16 at 2:17