The Bayes' Theorem Components of the Probability Output of a Classifier

Question

Let's give a simple setup.

I have $500$ photos of dogs and $500$ photos of cats, all labeled. From these, I want to build a classifier of photos.

For each photo, the classifier outputs a probability of being a dog (which I deem to be class $1$).

$$P(\text{dog }\vert\text{ photo})$$

We can reverse the conditioning with Bayes' theorem.

$$P(\text{dog }\vert\text{ photo}) = \dfrac{P(\text{photo }\vert \text{ dog}) P(\text{dog})}{P(\text{photo})}$$

I can interpret the $ P(\text{dog})$ as the prior probability of a photo being of a dog. Since the classes are balanced, I would call this $ P(\text{dog}) = 0.5$. Then the probability output of the classifier, $P(\text{dog }\vert\text{ photo})$ is the posterior probability of the photo being of a dog.

What are the interpretations of $P(\text{photo }\vert \text{ dog})$ and $P(\text{photo})?$

Each individually seems like it could be zero, so perhaps the better interpretation would be the ratio $\dfrac{P(\text{photo }\vert \text{ dog})}{P(\text{photo})}$.

In that case, what is the interpretation of the ratio? Either the ratio, or the numerator or denominator on its own, must have something to do with the particular model (e.g. convolutional neural network vs logistic regression), right?

Answer 1

I like the question.

One point before the explanation. In statistics, we use a capital letter $P$ for probability, as your prior. For probability densities a small letter $p$ is used.

The probability $P(photo \mid dog)$ assumes discrete input feature variables, associated with each photo. In image processing a 2-d image is represented as a grid of pixel values with $1$ intensity or $3$ colour intensity bands. It is most common to represent the pixel intensities (per band) as continuous distributions. In the one-band situation, $photo$ is an $r \times c$ matrix of pixel intensities. For convenience, $photo$ is mathematically often considered a vector. Its outcome is the pixel intensity distribution in the image, regardless of the spatial arrangement of the pixels. In the remaining answer the pixels are considered stochastic variables and their spatial arrangement is not taken into account.

Bayes rule

You can write Bayes rule as

$ \begin{split} P(dog \mid photo) =& &\frac{p(photo \mid dog) P(dog)}{p(photo \mid dog) P(dog) + p(photo \mid \neg dog) P(\neg dog)} \end{split} $

in which

$P(dog)=1-P(\neg dog)$. Clearly $\neg dog = cat$ in your setup.

Here $p(photo \mid dog)$ is an $n$-dimensional probability density function. If $p(photo \mid dog)$ does follow a normal distribution, then it's an $n$-dimensional normal distribution with the density

$ \begin{split} p({\bf x}; {\bf \mu}, \Sigma) = & \\ &\frac{1}{(2\, \pi)^{n/2} |\Sigma \mid^{0.5}} \cdot \exp \left(- \frac{1}{2}({\bf x}-{\bf \mu})^T \, \Sigma^{-1} ({\bf x}-{\bf \mu}) \right) \end{split} $

where ${\bf x}$ and ${\bf \mu}$ are both vectors and $\Sigma$ the symmetric covariance matrix.

Of course many different kinds of continuous distributions appear in practice and so the normal distribution is often ill-suited as representation. You can instead use for example the nonparametric kernel densities to model $p(photo \mid dog)$ and $p(photo \mid cat)$, based on the values of your training set.

The distribution $P(dog)$ is in any case a mixture distribution. This mixture has more 'peaks'

$ p(photo) = p(photo \mid dog) P(dog) + p(photo \mid cat) P(cat) $

In the situation where $p(photo \mid dog)$ and $p(photo \mid cat)$ are normal distributions, $p(photo)$ contains two 'peaks'. Note the two normal distributions can have very different variances. The 'widths' of the two distributions will then differ.

Interpretation

The fraction mentioned in the question above

$ \begin{split} \frac{P(photo \mid dog)}{P(photo)} = & & \frac{P(photo \mid dog)}{p(photo \mid dog) P(dog) + p(photo \mid cat) P(cat)} \end{split} $

is a likelihood ratio, but not the one which is applied in probabilistic decision analysis. Note that the prior probabilities occur in the denominator, but not in the numerator of the previous formula.

Instead, the class-conditional likelihood ratio

$ \begin{split} \mathcal{L}\mathcal{R} = \frac{p(photo \mid dog)}{{p(photo \mid \neg \, dog)}} \end{split} $

is used in probabilistic decision analysis. The $\mathcal{L}\mathcal{R}$ is independent of the prior distribution. It expresses the odds of a specific 'photo' belonging to the two categories. In the case where the prior probabilities are equal, then

$ \begin{split} \frac{p(photo \mid dog)}{{p(photo \mid \neg \, dog)}} =& &\frac{P(dog \mid photo )}{{P(\neg dog \mid photo )}} \end{split} $

In case of uneven priors, the prior probabilities $P(dog)$ and $P(\neg dog)$ form part of this equation as well.

Answer 2

I think defining the events well is the key here. Let's define them as:

$photo$: you choose that picture for classification.

$dog$: the picture is a dog.

Then $P(photo|dog)$ is the probability that you choose that specific photo for classification, given that you choose a picture of a dog, and $P(photo)$ is simply the probability that, out of all the photos, you choose that photo to use for classification.

For example, let's say you had 6 photos (5 dogs and 1 cat), and you're trying to classify photo #1, which is a dog (so that $P(dog|photo) = 1$).

$P(photo)$ is be $1/6$, since the probability of picking any photo is 1/6.

$P(photo|dog) = 1/5$, since if you choose a random photo of a dog, the probability it's photo #1 is 1/5.

and $P(dog) = 5/6$, since there's 5 dogs out of the 6 total photos.