# Likelihood of a random variable vs. Likelihood of a sample

As mentioned in the title, I am confused over the difference between $$L(\theta|S)$$ and $$L(\theta|X)$$, where $$X = (X_1, X_2, ... ,X_n$$).

From what I understand, $$L(\theta|S)$$ is the probability of $$\theta$$ being the true probability given a response $$S$$, which is the sample. This can be useful in a sense that the peak of $$\theta$$ can be shown through a graph. This peak represents the highest likelihood for the parameter $$\theta$$.

As for $$L(\theta|X = X_1, X_2, ..., X_n)$$, it implies the likelihood of $$\theta$$ given a set of different probability measures. This can be useful in the same way where an MLE can be derived using a set of steps (i.e. differentiating the score fn, etc.).

I'm not sure if I failed to understand the difference or if my textbook just didn't outline a difference. Or can they be used interchangeably?

p.s. In terms of using the peak of $$\theta$$, I'm trying to express my understanding of likelihood function, I know there are more ways to express it's usefulness.

• You can write the sample as $S=(X_1,X_2,\ldots,X_n).$ In light of this, doesn't your question reduce to one of orthography?
– whuber
Apr 8 '19 at 18:53
• Can you cite the text you are working from? Nowadays anyone can get LaTeX and upload technical-looking documents that are wrong. Some parts of your question are incorrect/unclear. If you are working from a good text (Casella & Berger, Hogg & Craig, etc.), we can show you where to read more. Apr 8 '19 at 19:21
• yes, this is a bad question. I think I understand what I was confused about earlier though. Apr 8 '19 at 19:42

## 1 Answer

The likelihood is not a probability. It's pretty easy to understand from a simple example; sample 3 heights, they are 1.3m, 1.6m and 1.9m. The "mean" height is 1.6m and, if those heights are normal, that maximizes the likelihood. However, average height is not a random quantity, the sample outcomes are. if I did the study again, the probability of sampling exactly 1.3m, 1.6m, and 1.9m is exactly 0, as is the probability of sampling 2.3m, 2.6m, and 2.9m: height is continuously valued.

S is not a response or a sample in most notations, it is a sufficient statistic. Another notation for sufficient statistic is $$T$$ when $$S$$ is reserved to mean a sum... although for many practical distributions (normal, bernoulli, poisson, etc.) the sum is a sufficient statistic. A sufficient statistic can be thought of as "summarizing" the whole sample (when it comes to parametric estimation). Any statistic independent of the sufficient statistic is called an ancillary statistic. A sufficient statistic effectively summarizes the likelihood for the whole sample, so that the MLE: $$\text{arg max}_\theta L(\theta, S_n) = \text{arg max}_\theta L(\theta, X_1, X_2, \ldots, X_n)$$.

• In terms of S, if there was a general question like $S = (X_1, X_2, ..., X_n)$ ~ $Ber(\theta)$ which asked me to find the mle of $\theta$, do I interpret the X's in S to be a sufficient statistic? If so, what's the intuition behind this that differs each X? I know the X's are iid so you can multiply them together to find the joint pdf, but what's the difference between each of these X's? Why doesn't the question just ask me to find the mle of $\theta$ given a sample where there are n samples of the same $X$~$Ber(\theta)$? Sorry if my wording is bad, I hope you can understand my confusinon Apr 8 '19 at 19:39
• @DonaldMayer No. The sum is sufficient for Bernoulli sample. That has a binomial distribution and finding an MLE is easy. The ancillary statistic is the hypergeometric distribution, showing the actual Xs that take 1 values. We purposely don't care about that. See here stat.wisc.edu/courses/st312-rich/suff2.pdf Apr 8 '19 at 19:46
• So I've just read the article you provided. I understand that a sufficient statistic is one that can reduce data such that it retains the largest amount of relevant info about the distribution. So if a question said something like "Given an unobservered sample of $S = (X_1, X_2, .., X_n)$ that are of distribution Y($\theta$), do we just assume S is a statistic? I thought it the X's in S were of a set of n probability measures with different non-parameter variable values (i.e. the small x in poisson distribution) Apr 8 '19 at 20:14
• You keep confusing $S$ as a sufficient statistic with $S$ as an unnecessary shorthand for the sample $S = X_1, X_2, \ldots, X_n$. Which is it? Indeed the whole sample is a statistic, and it is a sufficient statistic, but it is not a minimally sufficient statistic, because you can condition on an ancillary statistic. Apr 8 '19 at 20:43