Methods to weight probabilities that are being multiplied together?

Question

I am trying to classify data according to a taxonomy:

That is, given a feature vector $x$, I would like to compute:

$P(A,\alpha)$

$P(A,\beta)$

$P(B,\delta)$

$P(B,\gamma)$

(You can think of the above as four different class labels)

Due to the nature of the data, I am computing the above four probabilities as:

$P(A,\alpha) = P(A)P(\alpha|A)$

$P(A,\beta) = P(A)P(\beta|A)$

$P(B,\delta) = P(B)P(\delta|B)$

$P(B,\gamma) = P(B)P(\gamma|B)$

However, I know that my estimates of $P(A)$ and $P(B)$ are much more reliable than my estimates of $P(\alpha|A)$, $P(\beta|A)$, $P(\delta|B)$, and $P(\gamma|B)$.

Therefore, I would like to give $P(A)$ and $P(B)$ more weight in the four formulas above. What are good methods to do this?

(Also I would like to avoid $P(X,Y)$ from becoming super small due to some weighting mechanism)

Thanks in advance!

Answer 1

It seems like what you're moving toward is a Bayesian model. If your data is always binary as in your examples, then what you want to consider is putting a prior on $P(\alpha | A)$ most likely in the form of a beta distribution. Then having observed data, when you estimate $p = P(\alpha | A)$ you will take the expected value of the posterior distribution, which takes into account the data and the prior.

The beta distribution is essentially a coin factory, and it is the conjugate distribution to the binomial distribution. This simply means that the posterior distribution will also be a binomial distribution. [see edit below]

The simplest way of achieving this is through pseudo-counts. Essentially before seeing any data you could begin with two tallies for $\alpha$ and 4 tallies for $\beta$. This will push your estimate of $p$ toward 1/3, but actual data will have the power to move it away if need be. Starting with 200, 400 would make this force much harder to overcome.

If your data is not binary in nature but categorical, and you were just giving a simple example, then your distribution is multinomial and your prior should be Dirichlet. Again, this prior can be expressed in the form of pseudo-counts but this is less explicit than a specified prior distribution.

I know you wanted to give more weight to P(A) and this might not seem to do this, but you can think of your desire as wanting your estimates to be less volatile with respect to small amounts of data at the $\alpha$ level, and this will have that property. So $P(\alpha)$ will depend more on #A/total then it will on #$\alpha$/#A.

Edit: my summary of conjugate distributions was completely inverted. Conjugacy here means that beta prior on the parameter of a binomial distribution, will lead to a beta posterior, once data is incorporated.