I am studying about maximum likelihood estimation and I read that the likelihood function is the product of the probabilities of each variable. Why is it the product? Why not the sum? I have been trying to search on Google but I can't find any meaningful answers.
$\begingroup$
$\endgroup$
3
7 Answers
$\begingroup$
$\endgroup$
4
This is a very basic question, and instead of using formal language and mathematical notation, I will try to answer it at a level at which everybody who can understand the question can also understand the answer.
Imagine that we have a race of cats. They have a 75% probability of being born white, and 25% probability of being born grey, no other colors. Also, they have a 50% probability of having green eyes and 50% probability of having blue eyes, and coat color and eye color are independent.
Now let us look at a litter of eight kittens:
You will see that 1 out of 4, or 25%, are grey. Also, 1 out of 2, or 50% have blue eyes. Now the question is,
how many kittens have grey fur and blue eyes?
You can count them, the answer is one. That is, $\frac{1}{4} \times \frac{1}{2} = \frac{1}{8}$, or 12.5% of 8 kittens.
Why does it happen? Because any cat has a 1 in 4 probability to be grey. So, pick four cats, and you can expect one of them to be grey. But if you only pick four cats out of many (and get the expected value of 1 grey cat), the one which is grey has a 1 in 2 probability to have blue eyes. This means, of the total of cats you pick, you first multiply the total by 25% to get the grey cats, and then you multiply the selected 25% of all cats by 50% to get those of them which have blue eyes. This gives you the probability of getting blue-eyed grey cats.
Summing them up would give you $\frac{1}{4} + \frac{1}{2}$, which makes $\frac{3}{4}$ or 6 out of 8. In our picture, it corresponds to summing up the cats which have blue eyes with the cats which have grey fur - and counting the one grey blue-eyed kitten twice! Such a calculation can have its place, but it is rather unusual in probability calculations, and it is certainly not the one you are asking about.
-
1$\begingroup$ I am aware that the other answers here mean the same thing. Still I think that a visual representation is needed here - if the OP was able to visualize the concept himself, he probably would have arrived at the answer already. $\endgroup$– rumtschoMay 10, 2016 at 20:26
-
$\begingroup$ This is actually a terrific answer as it shows each independent variable as an independent axis in the cat matrix. This makes it very easy to understand. I'll be using this example to teach my children! $\endgroup$ May 11, 2016 at 9:42
-
3$\begingroup$ This answer is actually flawed, because it still conflates observed value and expected value. Seeing how popular it is, I will try finding the time to update it with an explanation why this way of subsetting the cats gives us a maximum likelihood estimator (or, resolving the problem of picking 8 random cats and finding out that they are not the ones I painted in the picture). $\endgroup$– rumtschoMay 11, 2016 at 10:41
-
$\begingroup$ Why can't this be the entire population of such cats? (Say they have some special research property -- their tongues are chemiluminescent, for instance.) Then the conflation is non-deleterious. $\endgroup$ May 11, 2016 at 21:59
$\begingroup$
$\endgroup$
2
Independence between two events means that the occurrence of one event does not affect the likelihood of the occurrence of the another event . So for any two events $A$ and $B$ in a sample space $S$ we say that $A$ and $B$ are independent iff $P(A$ and $B)=P(A\cap B) = P(A)P(B)$ .Now for more than two events we say that the events $A_1,A_2,...A_n $ are independent iff $P(\underset{i\in I}{\cap A_i})= \prod_{i\in I} P(A_i)$ for all subsets $I \subset [1,2,...,n]$ .
In the likelihood we suppose that there is a sample $x_1, x_2, …, x_n $of $n$ independent and identically distributed observations (iid), coming from a distribution with an unknown probability density function , that means this joint density function is $f(x_1,x_2,...,x_n|\theta) = \prod_{i=1}^{i=n}f(x_i|\theta)$.
answered May 10, 2016 at 16:48
-
$\begingroup$ The correct notation for the joint density function representing the likelihood wouldn't be $f(x_1,x_2,...,x_n;\theta) = \prod_{i=1}^{i=n}f(x_i;\theta)$? I guess this ";" is used to not confuse with the bayesian version. $\endgroup$ Jul 17, 2021 at 16:17
-
1$\begingroup$ So is it correct to assume that whenever we use maximum likelihood estimates, we are assuming iid for our observations? $\endgroup$– Shan DouOct 14, 2021 at 15:38
$\begingroup$
$\endgroup$
4
Under the common assumption of independence, $P(A \cap B)$ = $P(A) P(B)$.
Thus, if you assume that all your observations are independent, then the probability of observing all the values you saw is equal to the product of the individual probabilities.
-
10$\begingroup$ I think it would help the OP if you also explain why $P(A \cap B)$ is of interest here. $\endgroup$ May 10, 2016 at 16:17
-
$\begingroup$ Hi thanks for the reply ! Why do I maximize the likelihood (joint density function) ? Why can't I instead maximize the sum of the probabilities of all observation (or any other function) ? I would like to find the reason why the joint density function is chosen. Wikipedia starts off by using the joint density function. But is there a reason why we use the joint density function ? This is what I have been trying to understand. $\endgroup$– RuiQiMay 10, 2016 at 16:46
-
$\begingroup$ @haziqRazali the idea of MLE is to pick the estimates so as to the make the sample you have most likely given the distribution. Hence the name maximum likelihood $\endgroup$– RepmatMay 10, 2016 at 19:18
-
1$\begingroup$ @HaziqRazali A question like "why maximize likelihood" is a new question (one that has been asked and answered elsewhere on the site) $\endgroup$– Glen_bMay 10, 2016 at 23:17
$\begingroup$
$\endgroup$
Why not add?
Because that clearly makes no sense. Suppose you have a quarter and a nickel, and you want to flip them both. There's a 50% chance the quarter will come up heads, and a 50% chance the nickel comes up heads. If the chance of both coming up heads were the sum, that would make 100% chance, which is obviously wrong, as it leaves no chance for HT, TH, and TT.
Why multiply?
Because it does make sense. When you multiply the 50% chance of the quarter coming up heads by the 50% chance of the nickel coming up heads, you get 0.5 x 0.5 = 0.25 =25% chance of both coins being heads. Given that there are four possible combinations (HH, HT, TH, HT) and each is equally likely, this fit perfectly. When evaluating the likelihood of two independent events both occurring, we multiply their individual probabilities.
$\begingroup$
$\endgroup$
I am reading these posts because, like the Original Poster, my need is to understand why the 'Likelihood' fn is the 'Product' of the density of each sample value -'x'. A readable and logical reason is given under the heading Principle of maximum likelihood Ref: http://www-structmed.cimr.cam.ac.uk/Course/Likelihood/likelihood.html A further quotation Mathematically, the likelihood is defined as the probability of making the set of measurements (same ref.) In short, the probabilty that you arrived at the sample that you have at hand.
$\begingroup$
$\endgroup$
The goal of the maximum likelihood method is find estimator that maximize the probability of observe certains values of the variable ( endogenous variable). That is the reason why we must multiply the probabilties of ocurrence.
For example: imagine that the numbers of phone calls that a secretary can answer in an hour follows a poisson distribution. Then, you extract 2 values of the sample ( 5 phone calls and 8 phone calls per hour) Now you must answer this question. What is the value of the parameter that maximize the probability of observe 5 and 8 phone calls, simultaneously?. After, try to answer with the probaility of observe all the values of the sam
Due to the independant random variables,
f(y1 = 5 phone calls)*f(y2 = 8 phone calls) = ∏i f(y,θ) = L(θ,y1,y2)
Finally, try to answer, the probability of observe all the values of the sample.
$\begingroup$
$\endgroup$
In practice, when we want to find MLE we would usually take the log of the likelihood function. Then the product of density function or probability (depend on the data type) shall become sum.
As to the question why likelihood is a product of probability. Considering "The goal of maximum likelihood estimation is to find the values of the model parameters that maximize the likelihood function over the parameter space", we want to maximize the likelihood and a best estimate is the probability. Suppose there is a normal distribution with parameters $\mu$ and $\sigma^2$, response Y, $x_i$ are independent, then $L(\mu, \sigma^2 | Y) = P(y | \mu, \sigma^2)$. Because probability of independent events $P(A\cup{B}) = P(A)P(B)$. The likelihood becomes the product of the probability.
80% + 75% = 155%? how about80% * 75% = 60%? $\endgroup$