1. Many define the likelihood of the data something like $\prod_{x} p(x|\theta)$ others like $p(x|\theta)$. Is the likelihood defined for one sample point/data element (like one document from a collection of documents or one sentence from a collection of sentences), for the whole collection of data elements, or for both?

  2. Is the likelihood, in the Expectation maximization (EM) algorithm $L(\theta|X)$ https://en.wikipedia.org/wiki/Expectation%E2%80%93maximization_algorithm#Description and in the Maximum Likelihood Estimation (MLE) algorithm https://en.wikipedia.org/wiki/Maximum_likelihood_estimation#Principles, considered to be taken from one data element or from the whole collection of data elements? By the way, in the EM link why the likelihood is written as $L(\theta|X)=P(X|\theta)$ I know that capital X means a random variable in probability, but likelihood is defined on the data so they should write either $P(x|\theta)$ or $P(X=x|\theta)$?

  • 2
    $\begingroup$ Possible duplicate of What is the difference between "likelihood" and "probability"? $\endgroup$
    – Noah
    Apr 3 '19 at 16:17
  • $\begingroup$ @Noah no it is not a duplicate, I already read that article. $\endgroup$ Apr 3 '19 at 16:19
  • $\begingroup$ It at least answers 3 and 4. $\endgroup$
    – Noah
    Apr 3 '19 at 16:21
  • $\begingroup$ @Noah No, my question is completely different.... $\endgroup$ Apr 3 '19 at 19:37
  • 2
    $\begingroup$ Because you are still asking exactly the same questions about the meanings of the symbols and the relationships between the likelihoods and probabilities. $\endgroup$
    – whuber
    Apr 4 '19 at 19:51

A short answer that works in general: these terms are sort of loose, and can always mean whatever is the most useful in a particular context. If you have a whole bunch of samples, why not use them all and consider $X$ to be the collection of all samples. If you have only a single sample, then that's all you can use. You are free to do whatever you want.

But to answer your questions more directly:

  1. You give two expressions for likelihood: $\prod_x p(x\mid\theta)$ and $p(x\mid\theta)$. The second expression is just the conditional probability of a point $x$ given a choice of parameters $\theta$, and that's how it should be interpreted. The former expression is the product of many of these conditional probabilities, or in other words it is the conditional probability of many datapoints $X=\{x\}$ assuming that they are independent. In general, if we didn't know anything about independence of the datapoints, we would have to work with the joint distribution $p(X\mid\theta)$, but given that $x$ are independent we can factor the distribution into the product $\prod_x p(x\mid\theta)$. The independence assumption is common so that's probably why you saw it.

  2. I hinted at this in 1, but people often use lowercase $x$ to refer to a single sample, and capital $X$ to refer to the whole collection of data in this context of likelihoods of data. (Often in a more theoretical context a capital $X$ would be a random variable; the capital here comes from the convention that matrices are capital letters.) So as a rule of thumb I'd say $L(\theta\mid X)$ would probably be the likelihood of the parameters given a whole dataset. But again, you are free to do what you want.

  3. Again, it is not necessarily true that capital $X$ means a random variable. It might be true in that context, but you should always pay attention to the context and the usage of the authors you're interested in. Mathematicians are loose with notation and that is what it is. I guess my moral here is not to always assume that the same letters mean the same things in different papers. For instance, on that EM page, they are doing something rather complicated, which is taking an expectation against the probability measure of the entire dataset. This object is rather abstract and hard to conceive of, and I think it's best to understand EM concretely by working through a particular case -- it's really a whole family of algorithms and describing it in general tends to be vague.

Answers to questions in comments:

  • I can say that for sure, $P(X=x\mid\theta)$ tends to refer to the probability that a random variable $X$ takes on a value $x$ given the parameters $\theta$. But in $P(X\mid\theta)$, $X$ might be a random variable or a dataset. Again, these things should always be clear in a given case, so I think it's not worth worrying about these generalities. In the case where $X$ is a random variable, $P(X\mid\theta)$ is probably referring to the entire distribution rather than a particular probability, so I might be inclined to interpret it as a probability distribution (a function(al)) rather than a probability (a number) if I came across it.

  • As for the notation on Wikipedia that you asked about -- certainly it can be both, and MLEs will improve in accuracy with more data, so you might as well think about it as the whole dataset. But Wikipedia tends to have really strange and inconsistent notation since it is written collaboratively by random people with different backgrounds, so I would really not stress the notation on that site. In particular, if you're trying to learn these things, don't do it on Wikipedia -- get a textbook. Maybe "Elements of Statistical Learning" or another classic text -- I think that's outside the scope of this question.

  • $\begingroup$ thanks. Can we say that if it is written like $P(X|\theta)$ then X means the data set and when it is written like $P(X=x|\theta)$ then X means a random variable? Also suppose someone declares X as a random variable, is it still correct to say $P(X|\theta)$ or it is wrong because when we mention X we must assign a value to x? $\endgroup$ Apr 3 '19 at 19:23
  • $\begingroup$ Can you do a favor also and tell me what is x in the Maximum Likelihood Estimation en.wikipedia.org/wiki/Maximum_likelihood_estimation#Principles when they wrote $L(\theta;x)$, is it one sample, the whole data set, or can be both but better to use the whole set? $\endgroup$ Apr 3 '19 at 19:33
  • $\begingroup$ Sure -- I've edited my answer to include responses to these questions since I wanted to say more than would fit here. $\endgroup$
    – cwindolf
    Apr 3 '19 at 19:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.