Beginner: Understanding difference between pmf, conditional pmf and likelihood

Question

I have a point of confusion regarding the three types functions. I have looked at some other posts here and blogs and scripts and YouTube videos. But I still don't get it.

Let's look at the coin toss experiment.

When the question is:

The probability for heads is 0.7 and I toss a coin 7 times, what is the probability of heads coming up 5 times?

Then I do this calculation using the Binomial distribution:

$B(5|0.7,7) = \binom{7}{5}0.7^50.3^2$

When I use Bayesian inference to determine the probability for heads in a series of coin tosses, I use the Binomial distribution as my model (I'll put k instead of the binomial factor)

So, $P(5|7,\theta) = k \theta^5(1-\theta)^2$

My confusion is this: The formula for the likelihood looks to me like the formula in the first case.

In the first case, I have an unconditional PMF, right? But when I read about Bayesian inference, the model is always described as a conditional probability (or at least the term "conditioning" is used and then it is pointed out that likelihoods aren't probabilities). But why then is the formula in both cases the same?

Or is the first already a conditional probability?

It seems to me like the formula for the likelihood is always just the PMF or PDF and the difference is in how it is interpreted. Is that correct?

I guess what I am saying is this: I dont see the difference between the mathematical expression for p(x) in examples like the first one, where the task is to calculate a concrete probability, and p(x|\theta) as it is used in the coin toss examples that try to explain bayesian inference.

But from my understanding, they are not the same thing, they are not both formulas for PMF. And if so, why is the term "conditional" used in one context?

I hope I kinda got my point across.

Edited question

Thanks Stataphobia for your answer, and for the comments!

So, yeah, the fact that the notation is not used consistently does not make it easier for someone like me. Wouldn't it solve the problem if $P(X|\theta)$ was only used for actual conditional probabilities and in all other cases you either leave out the parameters or use a semicolon, like $P(X;\theta$)?

So let me see if I understood what you said about PMF vs. likelihood: Yes, they have the same form (or yes, we do use the PMF/PDF), but they are interpreted differently and the likelihood is not a conditional probability, even though we are conditioning on something (the data).

And the PMF may already be considered a conditional probability (so the notation with the | is justified), but often is not, specifically when $\theta$ is not a random variable (which in the frequentist view it never is, as far as I understand).

So back to the coin toss example: if I understand correctly, in the frequentist case, we use the formula for the PMF to calculate $P(X)$ and the likelihood $L(\theta|x)$.

In the bayesian case, it is also the formula for the PMF that is used for the likelihood, but here the PMF is considered an already conditional PMF, because $\theta$ is regarded a random variable.

So I guess my big mistake is to somehow expect the formula for the PMF and the conditional PMF to look different. But I guess this again comes from the fact that I mistakenly think the likelihood is a conditional probability.

I think it just confuses me that the formula for the PMF can simply be used as a conditional PMF, simply by conceptionally regarding one of its parameters as a random variable.

Maybe this shows what I mean (am not sure if this scenario even makes sense): Let's say I have a random varible X, distributed binomially and a r.v. Y, distributed normally. Now I want to calculate $P(X|Y) = \frac{P(X,Y)}{P(Y)}$, not in the context of inference. Here, in the numerator, I am multiplying the binomial PMF with the normal PDF, right? And then dividing by the binomial PMF. `Will the resulting formula in the right look like the PMF of the binomial?

But I guess this is different from what happens when calculating the likelihood in the Bayesian inference: There, I am not really calculating $P(X|\theta) = \frac{P(X,\theta)}{P(\theta)}$. I am using the formula for the PMF and it is just called a conditional PMF.

When you look at those beginners exercises like: You throw a dice, whats the probability that it shows an even number? Well, $\frac{3}{6}$ for a fair dice. Whats the probability is shows an even number given it is less than 5? Thats the conditional probability P(is even|less than five), so $\frac{\frac{3}{6}*\frac{4}{6}}{\frac{4}{6}}$

This looks to me like there is a clear distinction between P(X) and P(X|Y), conceptionally and how it is calculated. And I kind of expect this to be reflected in the formulas for the PMF and conditional PMF.

And one more question: In your last statement you say, you kind of think of all probabilities as conditioned on something. But we do distinguish between conditional and marginal, right? So does that mean that the marginal can also be somehow viewded as a conditional?

Answer 1

Including my original answer at the bottom below the line. I may have confused you by including too many of the subutilies about conditional probability.

First, I would suggest that if you are learning about these things for the first time that you focus on learning about the frequentist approach first and wait till you have mastered that before trying to learn about the Bayesian approach. The frequentist approach is a little easier to learn, and you won't be able to understand the differences between the two until you learn that.

At this point it is easier to start with simple rules about what is and what is not conditional probability before learning about the subtleties and situations where the distinction can become a bit blurred. In the frequentist world all parameters are viewed as fixed so the simplest distinction between conditional and unconditional probability is that when a probability has all terms on the right side of the conditioning bar | are parameters (usually represented by greek letters) then it is just probability (also called marginal probability or unconditional probability, but only when it is important to distinguish it as being different from conditional probability). When any of the terms on the right are random variables then it is a conditional probability.

$P(Y=y|\theta)$ probability

$P(Y=y|X=x,\theta)$ conditional probability.

For conditional probability we always need the term being conditioned on written to the right of the conditioning bar, but for probability (i.e. unconditional probability) there is actually quite a bit of variety in how you will see the notation in a book. Sometimes as $P(Y=y|\theta)$ will be written as $P(Y=y;\theta)$ and other times as $P(Y=y)$, but all mean the same thing, really just depends on the situation which notation is best.

A PMF (or PDF) is just a probability. It can either be a marginal probability or a conditional probability, but whichever it is for a given random variable $X$ there are not two different versions: one marginal and one conditional. Either you want the probability of $X$ unconditional and you have a PMF or you want it conditional on something and they you would use a different PMF, (but in either case there is only one PMF)

unconditional PMF: $P(X=x|\theta)$ there is not a 'conditional version' of this

You could consider a different situation where you want to condition on say a variable $Y$ then $P(X=x|Y=y,\theta)$ is a conditional PMF

$P(X=x|\theta)$ and $P(X=x|Y=y,\theta)$ are not two different versions of the PMF; They are just two different PMFs for two different situations.

The likelihood function is always equal to the PMF/PDF, but with a different interpretation. It is not a conditional probability. And although the functions look the same we do view it differently and we don't really view it as a probability at all. Just a function of the parameter.

Again, to simplify things you should really just be focused now on learning about PMFs and PDF and how conditional probability works and hold off on learning about likelihood functions, which are a more advanced topic.

---

I think there are two reasons for your confusion and it related to two somewhat different things. One having to do with the term conditional probability and one having to do with the likelihood function. Let's look at the first, so forget for the moment this having anything to do with likelihood functions.

As a start you need to know that not everyone uses the distinction between a 'conditional probability' and a 'probability' in the same way. Let's look at the binomial probability mass function (PMF)

$P(X=x|\theta)=$ $n\choose x$$\theta^x(1-\theta)^{n-x}$

The question is do we call $P(X=x|\theta)$ a conditional probability or not? The answer is that depends on who you ask. It sure looks like a conditional probability because it has a conditioning bar $|$, so I would call that a conditional probability (specifically, conditional on $\theta$). But many others would not, the reason being that when that formula is used to calculate a probability you are always using a fixed value for $\theta$ as in your example where $\theta=0.7$.

$P(X=x|0.7)=$ $n\choose x$$0.7^x(1-0.7)^{n-x}$

and some people prefer to only use the term 'conditional probability' when you are conditioning on a random variable.

It is common practice (especially among non-Bayesians) when a probability depends on only a fixed value not to call it a conditional probability. And to instead only call something a conditional probability when you are conditioning on something that is random. So, if for example we had a second random variable $Y$ and we calculate the probability of some value of $x$ conditional on some value of $y$ as $P(X=x|Y=y)$ then that would be called a conditional probability because $Y$ is random. Note also, that when people write a PMF they do not always include the $|\theta$ part, so you will sometimes see it as

$P(X=x)=$ $n\choose x$$\theta^x(1-\theta)^{n-x}$

Note that I did not include $n$ in my notation above for $P(X=x|\theta)$, so did not write $P(X=x|n,\theta)$. (lots of times authors will not include $n$ in the notation. Again because when you use this formula $n$ is always a fixed value so the same logic applies with some calling that something being conditioned on and other not.

Now let's look at the likelihood part of your question. You use the PMF when you know the value of $\theta$ and want to calculate a probability, as you did to find the probability of $X$ being 5 when $\theta=0.7$. You only use the likelihood function when you do not know the value of $\theta$ and you are trying to estimate it from some data you have observed. So if you did not know the value of $\theta$ and flipped a coin say 100 times and got heads 23 times and wanted to then estimate $\theta$, now is where you use the likelihood function.

And yes we construct the likelihood function from the PMF, but we use different notation because we have a different purpose here (to estimate $\theta$ not to use a known value of $\theta$ to calculate a probability). The likelihood function is

$L(\theta|x)=$$n\choose x$$\theta^x(1-\theta)^{n-x}$

where $x$ is the data, so to use this likelihood you plug in the value of $x$ (and $n$)

$L(\theta|x)=$$100\choose 23$$\theta^{23}(1-\theta)^{100-23}$

and then you maximize the function with respect to $\theta$ to estimate $\theta$ which is very easy to show gives an estimate 23/100.

A few things about the likelihood function.

1. It is not explicitly a Bayesian inference function. (used in both Bayesian and non-Bayesian inference).
2. We flip the ordering the $\theta$ and $x$ in the conditioning notation (with $\theta$ on the left side of the conditioning bar and $x$ on the right) because we are thinking of this as a function of $\theta$ conditional on $x$ with $\theta$ being the value we do not know and $x$ being the value we do know. (opposite of the situation where we know the value of $\theta$ and we use the PMF to estimate a probability of $x$ being some value)
3. while the likelihood is always equal to the PMF (or PDF) technically it is any function that is proportional with respect to the parameter $\theta$ to the PMF.

$L(\theta|x)\propto\theta^x(1-\theta)^{n-x}$

this is because $n\choose x$ does not contain $\theta$ so if you maximize this function with respect to $\theta$ you get the same answer.

There is quite a bit more to explain if you wanted to know how a Bayesian inference would use the likelihood function, but it does not appear to me that really is your questions, and additional details about it would not be helpful.

As a last thought here. My perspective is that ALL probabilities are conditional. There is always something that any probability depends on.

Answer 2

The probability is the area (integral) under the PMF or PDF function, the likelihood is the value of the PMF or PDF function about the parameters.

Bayesian inference uses data to make inferences about a parameter $\theta$, $$f(\theta | x^n) = \frac{f(x^n | \theta) f(\theta)}{\int f(x^n | \theta) f(\theta) d\theta}$$ here the $f(\theta | x^n)$ is the likelihood.

Let $C = (a, b)$, the conditional probability (or posterior interval) is an integral $$\mathbb{P}(\theta \in C | x^n) = \int_a^b f(\theta | x^n) d\theta$$