38
$\begingroup$

I am studying about maximum likelihood estimation and I read that the likelihood function is the product of the probabilities of each variable. Why is it the product? Why not the sum? I have been trying to search on Google but I can't find any meaningful answers.

https://en.wikipedia.org/wiki/Maximum_likelihood

$\endgroup$
3
  • 7
    $\begingroup$ Note that this is not necessarily the case, and in general the maximum likelihood is defined in terms of the joint density of the random variables. Of course if they are independent their joint density is just the product of the marginals $\endgroup$
    – Ant
    Commented May 10, 2016 at 21:38
  • $\begingroup$ Remember that multiplying is just a shorthand for addition. When I say 2 times 3 I'm saying 2+2+2. We multiply because we're lazy. Who has time to do it the hard way? You can add if it helps you see what's going on (helped me understand the Monty Hall problem) but after a while you'll get bored with it. $\endgroup$ Commented May 11, 2016 at 6:46
  • 1
    $\begingroup$ say you have a 80% probability of having brown hair, and a 75% probability of having brown eyes. Do you think it is possible that the probability of being brown-haired and brown-eyed is 80% + 75% = 155%? how about 80% * 75% = 60%? $\endgroup$
    – njzk2
    Commented May 11, 2016 at 22:05

7 Answers 7

53
$\begingroup$

This is a very basic question, and instead of using formal language and mathematical notation, I will try to answer it at a level at which everybody who can understand the question can also understand the answer.

Imagine that we have a race of cats. They have a 75% probability of being born white, and 25% probability of being born grey, no other colors. Also, they have a 50% probability of having green eyes and 50% probability of having blue eyes, and coat color and eye color are independent.

Now let us look at a litter of eight kittens:

enter image description here

You will see that 1 out of 4, or 25%, are grey. Also, 1 out of 2, or 50% have blue eyes. Now the question is,

how many kittens have grey fur and blue eyes?

You can count them, the answer is one. That is, $\frac{1}{4} \times \frac{1}{2} = \frac{1}{8}$, or 12.5% of 8 kittens.

Why does it happen? Because any cat has a 1 in 4 probability to be grey. So, pick four cats, and you can expect one of them to be grey. But if you only pick four cats out of many (and get the expected value of 1 grey cat), the one which is grey has a 1 in 2 probability to have blue eyes. This means, of the total of cats you pick, you first multiply the total by 25% to get the grey cats, and then you multiply the selected 25% of all cats by 50% to get those of them which have blue eyes. This gives you the probability of getting blue-eyed grey cats.

Summing them up would give you $\frac{1}{4} + \frac{1}{2}$, which makes $\frac{3}{4}$ or 6 out of 8. In our picture, it corresponds to summing up the cats which have blue eyes with the cats which have grey fur - and counting the one grey blue-eyed kitten twice! Such a calculation can have its place, but it is rather unusual in probability calculations, and it is certainly not the one you are asking about.

$\endgroup$
4
  • 1
    $\begingroup$ I am aware that the other answers here mean the same thing. Still I think that a visual representation is needed here - if the OP was able to visualize the concept himself, he probably would have arrived at the answer already. $\endgroup$
    – rumtscho
    Commented May 10, 2016 at 20:26
  • $\begingroup$ This is actually a terrific answer as it shows each independent variable as an independent axis in the cat matrix. This makes it very easy to understand. I'll be using this example to teach my children! $\endgroup$
    – dotancohen
    Commented May 11, 2016 at 9:42
  • 3
    $\begingroup$ This answer is actually flawed, because it still conflates observed value and expected value. Seeing how popular it is, I will try finding the time to update it with an explanation why this way of subsetting the cats gives us a maximum likelihood estimator (or, resolving the problem of picking 8 random cats and finding out that they are not the ones I painted in the picture). $\endgroup$
    – rumtscho
    Commented May 11, 2016 at 10:41
  • $\begingroup$ Why can't this be the entire population of such cats? (Say they have some special research property -- their tongues are chemiluminescent, for instance.) Then the conflation is non-deleterious. $\endgroup$ Commented May 11, 2016 at 21:59
22
$\begingroup$

Independence between two events means that the occurrence of one event does not affect the likelihood of the occurrence of the another event . So for any two events $A$ and $B$ in a sample space $S$ we say that $A$ and $B$ are independent iff $P(A$ and $B)=P(A\cap B) = P(A)P(B)$ .Now for more than two events we say that the events $A_1,A_2,...A_n $ are independent iff $P(\underset{i\in I}{\cap A_i})= \prod_{i\in I} P(A_i)$ for all subsets $I \subset [1,2,...,n]$ .

In the likelihood we suppose that there is a sample $x_1, x_2, …, x_n $of $n$ independent and identically distributed observations (iid), coming from a distribution with an unknown probability density function , that means this joint density function is $f(x_1,x_2,...,x_n|\theta) = \prod_{i=1}^{i=n}f(x_i|\theta)$.

$\endgroup$
2
  • $\begingroup$ The correct notation for the joint density function representing the likelihood wouldn't be $f(x_1,x_2,...,x_n;\theta) = \prod_{i=1}^{i=n}f(x_i;\theta)$? I guess this ";" is used to not confuse with the bayesian version. $\endgroup$ Commented Jul 17, 2021 at 16:17
  • 1
    $\begingroup$ So is it correct to assume that whenever we use maximum likelihood estimates, we are assuming iid for our observations? $\endgroup$
    – Shan Dou
    Commented Oct 14, 2021 at 15:38
7
$\begingroup$

Under the common assumption of independence, $P(A \cap B)$ = $P(A) P(B)$.

Thus, if you assume that all your observations are independent, then the probability of observing all the values you saw is equal to the product of the individual probabilities.

$\endgroup$
4
  • 10
    $\begingroup$ I think it would help the OP if you also explain why $P(A \cap B)$ is of interest here. $\endgroup$ Commented May 10, 2016 at 16:17
  • $\begingroup$ Hi thanks for the reply ! Why do I maximize the likelihood (joint density function) ? Why can't I instead maximize the sum of the probabilities of all observation (or any other function) ? I would like to find the reason why the joint density function is chosen. Wikipedia starts off by using the joint density function. But is there a reason why we use the joint density function ? This is what I have been trying to understand. $\endgroup$
    – RuiQi
    Commented May 10, 2016 at 16:46
  • $\begingroup$ @haziqRazali the idea of MLE is to pick the estimates so as to the make the sample you have most likely given the distribution. Hence the name maximum likelihood $\endgroup$
    – Repmat
    Commented May 10, 2016 at 19:18
  • 1
    $\begingroup$ @HaziqRazali A question like "why maximize likelihood" is a new question (one that has been asked and answered elsewhere on the site) $\endgroup$
    – Glen_b
    Commented May 10, 2016 at 23:17
5
$\begingroup$

Why not add?

Because that clearly makes no sense. Suppose you have a quarter and a nickel, and you want to flip them both. There's a 50% chance the quarter will come up heads, and a 50% chance the nickel comes up heads. If the chance of both coming up heads were the sum, that would make 100% chance, which is obviously wrong, as it leaves no chance for HT, TH, and TT.

Why multiply?

Because it does make sense. When you multiply the 50% chance of the quarter coming up heads by the 50% chance of the nickel coming up heads, you get 0.5 x 0.5 = 0.25 =25% chance of both coins being heads. Given that there are four possible combinations (HH, HT, TH, HT) and each is equally likely, this fit perfectly. When evaluating the likelihood of two independent events both occurring, we multiply their individual probabilities.

$\endgroup$
2
$\begingroup$

I am reading these posts because, like the Original Poster, my need is to understand why the 'Likelihood' fn is the 'Product' of the density of each sample value -'x'. A readable and logical reason is given under the heading Principle of maximum likelihood Ref: http://www-structmed.cimr.cam.ac.uk/Course/Likelihood/likelihood.html A further quotation Mathematically, the likelihood is defined as the probability of making the set of measurements (same ref.) In short, the probabilty that you arrived at the sample that you have at hand.

$\endgroup$
0
$\begingroup$

The goal of the maximum likelihood method is find estimator that maximize the probability of observe certains values of the variable ( endogenous variable). That is the reason why we must multiply the probabilties of ocurrence.

For example: imagine that the numbers of phone calls that a secretary can answer in an hour follows a poisson distribution. Then, you extract 2 values of the sample ( 5 phone calls and 8 phone calls per hour) Now you must answer this question. What is the value of the parameter that maximize the probability of observe 5 and 8 phone calls, simultaneously?. After, try to answer with the probaility of observe all the values of the sam

Due to the independant random variables,

f(y1 = 5 phone calls)*f(y2 = 8 phone calls) = ∏i f(y,θ) = L(θ,y1,y2)

Finally, try to answer, the probability of observe all the values of the sample.

$\endgroup$
0
$\begingroup$

In practice, when we want to find MLE we would usually take the log of the likelihood function. Then the product of density function or probability (depend on the data type) shall become sum.
As to the question why likelihood is a product of probability. Considering "The goal of maximum likelihood estimation is to find the values of the model parameters that maximize the likelihood function over the parameter space", we want to maximize the likelihood and a best estimate is the probability. Suppose there is a normal distribution with parameters $\mu$ and $\sigma^2$, response Y, $x_i$ are independent, then $L(\mu, \sigma^2 | Y) = P(y | \mu, \sigma^2)$. Because probability of independent events $P(A\cup{B}) = P(A)P(B)$. The likelihood becomes the product of the probability.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.