1
$\begingroup$

In Wikepedia it states that:

In the simplest cases, normalization of ratings means adjusting values measured on different scales to a notionally common scale, often prior to averaging.

And the likelihood funciton:
$\mathcal{L}(\theta; x_1,...,x_n) = f(x_1,...,x_n; \theta) = \prod_{j} f(x_j; \theta)$

As asked in this answer and this question, likelihood function is a function of the parameter only, with the data held as a fixed constantfunction and is not a pdf.

And in this tutorial of EM, he normalize the pair of likelihoods of a binomial distribution into a probability distribution simply by mean. I thought the two probabilities can be viewed as how likely the sample is from A or from B.

I wonder if there are other methods to normalize the likelihoods to scale them into numbers between 0 and 1. And what these two normalized likelihood stands for. Am I right that they represent how likely the sample is generated by coin A or coin B?

$\endgroup$

1 Answer 1

1
$\begingroup$

I think this is a bit of a misconception. There is no reason, in theory, why likelihoods are not normalised. They are normalised however w.r.t the data rather than the parameters. Let's say your target variable is given by y, then the likelihood is given by:

$\mathcal{L}= P(\underline{y}|\theta) = \prod_{i=1}^{N}P(y_{i}|\theta)$

If this target variable is binary, and can take the values 0 and 1, then the following must hold:

$\sum_{y_{1}=0,1}\sum_{y_{2}=0, 1}\ldots \sum_{y_{N-1}=0,1}\sum_{y_{N}=0,1}\prod_{i=1}^{N}P(y_{i}|\theta)=1$

So yes, you are right that the likelihood is a function of the parameters only, because the data is fixed, but the likelihood is still normalised w.r.t all possible values the data could take.

The posterior on the other hand, is normalised w.r.t to the parameters. In this notation:

$P(\theta| \underline{y})= \frac{P(\theta)P(\underline{y}|\theta)}{P(\underline{y})}$

While $P(\underline{y}|\theta)$ is normalised w.r.t to all possible data outcomes, multiplying it by $\frac{P(\theta)}{P(\underline{y})}$ means it is now normalised w.r.t the parameter.

Note that if we integrate both sides of the above w.r.t $\theta$, the RHS must integrate to 1, because given any data, our confidence that the parameters take some value must be 1, i.e. $\int P(\theta|D)d\theta =1$.

This means that $\frac{1}{P(\underline{y})}\int P(\theta)P(\underline{y}|\theta)d\theta=1$ or

$P(\underline{y})=\int P(\theta)P(\underline{y}|\theta)d\theta$

Now without watching the entirety of the video you posted above, it looks like the calculation in that video is asking you to evaluate the probabilities that a sequence of events was generated by distribution A or distribution B. Note that the parameters of distributions A and B are fixed (for this sub part of the problem). So you're not being asked to calculate a posterior, the pdf that some $\theta$ takes a range of continuous values. You're being asked to evaluate whether one or another hypothesis is true, given that one and only one must be true.

This is asking you to calculate $P(A|\underline{y})$ and $P(B|\underline{y})$, given by $\frac{P(\underline{y}|A)P(A)}{P(\underline{y})}$ and $\frac{P(\underline{y}|B)P(B)}{P(\underline{y})}$ respectively. It looks like an equal prior was used, so $P(A)=P(B)=\frac{1}{2}$.

In this case, $P(\underline{y})=\frac{1}{2}P(\underline{y}|A)+\frac{1}{2}P(\underline{y}|B)$ (i.e. because your hypothesis space is discrete, you sum rather than integrate as I claimed above), and thus

$P(A|\underline{y})=\frac{P(\underline{y}|A)\frac{1}{2}}{\frac{1}{2}P(\underline{y}|A)+\frac{1}{2}P(\underline{y}|B)}=\frac{P(\underline{y}|A)}{P(\underline{y}|A)+P(\underline{y}|B)}$

and similarly

$P(B|\underline{y})=\frac{P(\underline{y}|B)}{P(\underline{y}|A)+P(\underline{y}|B)}$

so in this case, he has "normalised" by dividing by $P(\underline{y}|A)+P(\underline{y}|B)$, but this is a nice simplification which arises because you're not trying to infer the value of a continuous parameter, you're trying to infer whether one of two things happened. If you were trying to infer the mean of a Bernoulli distribution, which can be between 0 and 1, you would need to calculate an integral. Furthermore, even in the case of two hypotheses, if the prior were not flat (i.e. you give me credence to hypothesis A or B), then the following would be true

$P(A|\underline{y})=\frac{P(\underline{y}|A)P(A)}{P(\underline{y}|B)P(B)+P(\underline{y}|B)P(B)}$ (and similarly B), so the normalisation would be different.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.