2
$\begingroup$

This question already has an answer here:

Let $X$ be a vector of sample data, and $W$ is a vector of parameters of a model based on that data, and we want to estimate a vector $W^*$ that is, informally speaking, "as close to $W$ as possible".

Then shouldn't we maximize $P(W^*|X)$? that is, we should choose the parameter estimates W* such that the probability that they are the actual parameters, is maximized.

Why then, in maximum likelihood, do we want to maximize $P(X|W^*)$ instead? I don't see the (intuitive) justification behind it.

(On a more general note, I also don't see why $P(X|W^*)$ is called "likelihood of $W^*$, given $X$", since "likelihood" and "probability" seem to me to be synonymous in the original meanings of the words).

$\endgroup$

marked as duplicate by kjetil b halvorsen, gung, Sycorax, COOLSerdash, Greenparker Aug 9 '16 at 17:48

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 3
    $\begingroup$ Does this help: stats.stackexchange.com/questions/224037/… ? TLDR; W is not a random variable, it does not have distribution, so we do not have conditional distribution for it unless you are willing to assume that it is a random variable and it has a distribution, but then you are a Bayesian. $\endgroup$ – Tim Aug 9 '16 at 10:15
  • 4
    $\begingroup$ Does this help: stats.stackexchange.com/questions/112451/… ? $\endgroup$ – Tim Aug 9 '16 at 10:33
  • 3
    $\begingroup$ Please explain to us how you would calculate P(W*|X) without invoking additional assumptions such as a prior distribution for W. $\endgroup$ – whuber Aug 9 '16 at 12:21
  • 1
    $\begingroup$ Likelihood & probability are not synonyms. See: Can a probability distribution value exceeding 1 be OK? & What is the difference between “likelihood” and “probability”? $\endgroup$ – gung Aug 9 '16 at 13:42
  • 2
    $\begingroup$ If I understand @programmer2134 correctly, this all really boils down to "if you're a frequentist, why not be a Bayesian instead". That doesn't seem to exactly duplicate anything suggested so far but I'm not sure how "answerable" that is. The best way I can see this question being "answered" is to restate Tim's first comment (if you're a frequentist, then W doesn't even have a distribution, so maximising P(W|X) is meaningless) and to perhaps to pick up on a few of the other misconceptions (eg as gung has done, when used as terms of art "probability" and "likelihood" are not synonyms) $\endgroup$ – Silverfish Aug 9 '16 at 13:48
5
$\begingroup$

"Likelihood" and "probability" in their colloquial usage are synonyms, but in statistics they have different meanings.

Let's start with defining our setting. We observed some independent and identically distributed datapoints $x_1,\dots,x_n$ and we have some model for this data that can be defined in terms of probability distribution $f$ parametrized by some unknown parameter $\theta$. Function $f_\theta(x_i)$ tells us about probability (or density) of observing $x_i$ value given some parameter value $\theta$.

Usually we calculate probabilities to check how probable is some $x$ given the distribution $f_\theta$ (e.g. what is the probability of obtaining $x$ points on some test knowing that the distribution of test scores is $f_\theta$). In case of likelihood function, we use it other way around and think about $f_\theta$ as of a loss function that returns higher values for $x$'s that better fit the distribution $f_\theta$ and lower values for $x$'s that are unlikely given this distribution.

To gain some more intuition, look at the plot below. It shows three probability density functions A, B and C and datapoints marked as red rug plot below the main plot. As you can see, in case of distribution A only a single point falls into highest density region, so this does not seem to be a good model. More points fall into highest density regions of functions B and C, but since B catches majority of points in it's highest density region, we can conclude that it fits the data better. This is very informal way of showing how likelihood works.

Different probability density functions against datapoints

More formally, likelihood function as a function of data given some fixed parameter that calculates total fit of $f_\theta$ across all the points

$$ \mathcal{L}(\theta|X) = P(X|\theta) = \prod_i f_\theta(x_i) $$

We call it likelihood since we do not care about actual probabilities and use the $f_\theta$ functions simply as loss functions. Second, in non-Bayesian setting $\theta$ is not a random variable, so we cannot compute $P(\theta|X)$, the direct answer for our question, so instead we use likelihood as a proxy.

Moreover, if we become Bayesians for a while and assume that $\theta$ is a random variable that follows continuous uniform distribution from $-\infty$ to $\infty$ (suppose this is the range of values that $\theta$ can take), then Bayes theorem reduces to calculating likelihood

$$ P(\theta|X) \propto P(X|\theta) P(\theta) = P(X|\theta) \times \text{const} \propto P(X|\theta)$$

and this is the same as maximizing the likelihood function over whole range of possible values of parameter

$$ \underset{\theta\in\Theta}{\operatorname{arg\,max}} ~ \mathcal{L}(X; \theta) $$

so it does not really make that much difference.

Check also:
Maximum Likelihood Estimation (MLE) in layman terms
What is the difference between "likelihood" and "probability"?
Wikipedia entry on likelihood seems ambiguous

$\endgroup$
  • $\begingroup$ The last 3 lines I think answer my question. However, why do you assume that P(theta) = const? $\endgroup$ – user56834 Aug 9 '16 at 14:36
  • $\begingroup$ @Programmer2134 because we assumed uniform distribution, so for every value from $-\infty$ to $\infty$ it returns the same probability density. $\endgroup$ – Tim Aug 9 '16 at 14:38
  • $\begingroup$ right, I understand, but why is this the assumption, since in by far most practical applications this does not apply, right? Or do frequentists acknowledge that maximizing likelihood only has relevance if we know little to nothing about the prior distribution of our parameters? $\endgroup$ – user56834 Aug 9 '16 at 14:50
  • $\begingroup$ @Programmer2134 in non-Bayesian settings there is no priors, so we do not assume a priori that any value of parameter is more likely than other. This example simply translates the frequentest setting to Bayesian perspective to show "what would happen" if in frequentist case we actually could compute $P(\theta|X)$. If we could, then it would not make any difference. And while we are talking about frequentist setting, there is no such a thing as a priori assumptions about parameters. $\endgroup$ – Tim Aug 9 '16 at 14:57

Not the answer you're looking for? Browse other questions tagged or ask your own question.