# The Bayes' Theorem Components of the Probability Output of a Classifier

Let's give a simple setup.

I have $$500$$ photos of dogs and $$500$$ photos of cats, all labeled. From these, I want to build a classifier of photos.

For each photo, the classifier outputs a probability of being a dog (which I deem to be class $$1$$).

$$P(\text{dog }\vert\text{ photo})$$

We can reverse the conditioning with Bayes' theorem.

$$P(\text{dog }\vert\text{ photo}) = \dfrac{P(\text{photo }\vert \text{ dog}) P(\text{dog})}{P(\text{photo})}$$

I can interpret the $$P(\text{dog})$$ as the prior probability of a photo being of a dog. Since the classes are balanced, I would call this $$P(\text{dog}) = 0.5$$. Then the probability output of the classifier, $$P(\text{dog }\vert\text{ photo})$$ is the posterior probability of the photo being of a dog.

What are the interpretations of $$P(\text{photo }\vert \text{ dog})$$ and $$P(\text{photo})?$$

Each individually seems like it could be zero, so perhaps the better interpretation would be the ratio $$\dfrac{P(\text{photo }\vert \text{ dog})}{P(\text{photo})}$$.

In that case, what is the interpretation of the ratio? Either the ratio, or the numerator or denominator on its own, must have something to do with the particular model (e.g. convolutional neural network vs logistic regression), right?

I like the question.

One point before the explanation. In statistics, we use a capital letter $$P$$ for probability, as your prior. For probability densities a small letter $$p$$ is used.

The probability $$P(photo \mid dog)$$ assumes discrete input feature variables, associated with each photo. In image processing a 2-d image is represented as a grid of pixel values with $$1$$ intensity or $$3$$ colour intensity bands. It is most common to represent the pixel intensities (per band) as continuous distributions. In the one-band situation, $$photo$$ is an $$r \times c$$ matrix of pixel intensities. For convenience, $$photo$$ is mathematically often considered a vector. Its outcome is the pixel intensity distribution in the image, regardless of the spatial arrangement of the pixels. In the remaining answer the pixels are considered stochastic variables and their spatial arrangement is not taken into account.

Bayes rule

You can write Bayes rule as

$$\begin{split} P(dog \mid photo) =& &\frac{p(photo \mid dog) P(dog)}{p(photo \mid dog) P(dog) + p(photo \mid \neg dog) P(\neg dog)} \end{split}$$

in which

$$P(dog)=1-P(\neg dog)$$. Clearly $$\neg dog = cat$$ in your setup.

Here $$p(photo \mid dog)$$ is an $$n$$-dimensional probability density function. If $$p(photo \mid dog)$$ does follow a normal distribution, then it's an $$n$$-dimensional normal distribution with the density

$$\begin{split} p({\bf x}; {\bf \mu}, \Sigma) = & \\ &\frac{1}{(2\, \pi)^{n/2} |\Sigma \mid^{0.5}} \cdot \exp \left(- \frac{1}{2}({\bf x}-{\bf \mu})^T \, \Sigma^{-1} ({\bf x}-{\bf \mu}) \right) \end{split}$$

where $${\bf x}$$ and $${\bf \mu}$$ are both vectors and $$\Sigma$$ the symmetric covariance matrix.

Of course many different kinds of continuous distributions appear in practice and so the normal distribution is often ill-suited as representation. You can instead use for example the nonparametric kernel densities to model $$p(photo \mid dog)$$ and $$p(photo \mid cat)$$, based on the values of your training set.

The distribution $$P(dog)$$ is in any case a mixture distribution. This mixture has more 'peaks'

$$p(photo) = p(photo \mid dog) P(dog) + p(photo \mid cat) P(cat)$$

In the situation where $$p(photo \mid dog)$$ and $$p(photo \mid cat)$$ are normal distributions, $$p(photo)$$ contains two 'peaks'. Note the two normal distributions can have very different variances. The 'widths' of the two distributions will then differ.

Interpretation

The fraction mentioned in the question above

$$\begin{split} \frac{P(photo \mid dog)}{P(photo)} = & & \frac{P(photo \mid dog)}{p(photo \mid dog) P(dog) + p(photo \mid cat) P(cat)} \end{split}$$

is a likelihood ratio, but not the one which is applied in probabilistic decision analysis. Note that the prior probabilities occur in the denominator, but not in the numerator of the previous formula.

$$\begin{split} \mathcal{L}\mathcal{R} = \frac{p(photo \mid dog)}{{p(photo \mid \neg \, dog)}} \end{split}$$

is used in probabilistic decision analysis. The $$\mathcal{L}\mathcal{R}$$ is independent of the prior distribution. It expresses the odds of a specific 'photo' belonging to the two categories. In the case where the prior probabilities are equal, then

$$\begin{split} \frac{p(photo \mid dog)}{{p(photo \mid \neg \, dog)}} =& &\frac{P(dog \mid photo )}{{P(\neg dog \mid photo )}} \end{split}$$

In case of uneven priors, the prior probabilities $$P(dog)$$ and $$P(\neg dog)$$ form part of this equation as well.

• I would note that p(photo|dog) is typically called the likelihood, and the denominator is typically called the evidence. Also, intuitively in this case, p(photo|dog) can be thought of as "given we have a photo of a dog, what is the probability we get this particular photo (arrangement of pixels)?" – eithompson Jul 27 at 19:40
• This interpretation doesn't ever tie to the model. Shouldn't it? – Dave Jul 27 at 19:56
• I adapted the answer to take into account your remarks - thanks! – Match Maker EE Jul 27 at 20:49
• It is rather extreme to model $p(\text{photo}\mid\text{dog})$ as a multivariate normal distribution when it is clear that photo is numerical valued, while dog and non-dog is Boolean-valued or Bernoulli at best – Dilip Sarwate Jul 28 at 17:07
• "It is rather extreme..." that is not the correct formulation. The fit will be bad ! - as I expect a multimodal distribution for each of the two te discern categories. Thanks ! – Match Maker EE Jul 28 at 17:10

I think defining the events well is the key here. Let's define them as:

$$photo$$: you choose that picture for classification.

$$dog$$: the picture is a dog.

Then $$P(photo|dog)$$ is the probability that you choose that specific photo for classification, given that you choose a picture of a dog, and $$P(photo)$$ is simply the probability that, out of all the photos, you choose that photo to use for classification.

For example, let's say you had 6 photos (5 dogs and 1 cat), and you're trying to classify photo #1, which is a dog (so that $$P(dog|photo) = 1$$).

$$P(photo)$$ is be $$1/6$$, since the probability of picking any photo is 1/6.

$$P(photo|dog) = 1/5$$, since if you choose a random photo of a dog, the probability it's photo #1 is 1/5.

and $$P(dog) = 5/6$$, since there's 5 dogs out of the 6 total photos.

• The classifier output won't be $1$. – Dave Jul 27 at 19:29
• @Dave, it will. It will be $(5/6*1/5)/(1/6) = 1$ – Nick Koprowicz Jul 27 at 19:31
• No, the convolutional neural network (or whatever) will output some probability. – Dave Jul 27 at 19:35
• This is just for illustration purposes. I thought the intention of your question was just about how to interpret the probabilities. You could change my numbers slightly just to make them not come out to 1, but that doesn't change the interpretations. – Nick Koprowicz Jul 27 at 19:38
• Yours is always going to result in $1$. – Dave Jul 27 at 19:45