# Confusion on Maximum Likelihood's starting setup equation

This is my first post on CV.

I am wrapping my head around Maximum Likelihood Estimation (MLE), and ironically, I understand everything about the algorithm, except the starting point. What confuses me is the following:

All the examples I have seen of course, start with some vector of observations, x1, x[2], ... x[N]. The observations are iid. The objective is to estimate parameters (theta) of the PDF - ONE PDF - from which sired such observations. This part I get. We are allowed to estimate ONE PDF because X ('big X') is a random variable from which all those x's (small x's) are sired.

As far as I know, big X's have PDFs. Small x's dont. Anyway, so the PDF whos theta we want to estimate is written like this:

p(x1, x[2], x[3]... x[n] | theta) = p(x1|theta) * p(x[2]|theta) * ... p(x[N]|theta).

Like I said if I accept this I know the rest of the algorithm, but this starting point is where I dont understand. Here all the sudden I now have N PDFs. This confuses me a lot:

Like I said before, as far as I know, observations (like x[5]) dont have PDFs. Random variables do. So in this case the observations are 'small x', where as a random variable is 'big X'. We are trying to estimate the pdf of 'big X' - X. How is it that I now have an individual PDFfor each and every observation 'small x'?

Every lecture video/article seem to use contradictory terms among themselves of 'random variable' and 'observation' while simultaneously taking the effort to point out that they are not the same and shouldnt be confused - so which one is it??

EDIT for whuber:

For additional context, here is my current state of understanding: By the way, I am going to assume a Gaussian process for X from here on out for this example.

• A random variable, usually denoted 'big something' - X in this case, can have a PDF. A 'small something' is the one outcome - one observation of big X, we call this 'small x'.

• So therefore, we can ask "What is the probability of X = 4?". Small x = 4 here, and this has meaning, and we can write "P(X=4) = ?". Big X's PDF evaluated at a chosen small x's value, gives us the probability X come out as x. In other words, P(X=x).

• To me this means only X can have a PDF. (X represents a name for a process that can generate many possible values - he can take many values), while x, cannot have a PDF. (What is the PDF of the number 7?). This is how I see it.

• If I have one observation, (x = 7), then I understand that the jist of the MLE is to say, which guassian, with what mu and sigma, maximize the probability of this one number being generated from it? If I had all the time in the universe, I would: 1) Take a guassian PDF with some mu and some sigma, and then 2) plug x = 7 into it and get a probability of it being 7. Eventually I would realize that the best mu for my one x = 7 is in fact when mu = 7.

• If I have two observations, (x1 = 7, x2 = 9), then my question in english is, "What possible combo of mu and sigma did my X have, such that I could have gotten x1 = 7 and x2 = 9 with the highest odds?". To me this means it is nonsensical to assume that x1 and x2 were aberrations and on the tail end of a gaussian. So not knowing anything else, we assume that well, if they are here, they were the most likely. So what values of mu and sigma most likely led to those being generated?

• This now means ok, what is the probability of x1=7 AND x2=9? So P(x1 = 7 AND x2 = 9) = P(x1=7) * P(x2=9). We can do this because they are IID.

EDIT 2

I have attached an image of what I believe to me a Likelihood function for ONE observation, (x = 3), assuming it came from a gaussian X, based on my new understanding from the convos here. Interesting...looks like the light pattern out of the headlight of a car...

• Just to see what's causing the difficulties, could you tell us whether you have any similar difficulties (with concepts, terminology, or computation) when there is exactly one observation, $x$? How about when there is exactly one observation which happens to be vector-valued? BTW, welcome to our community!
– whuber
Commented Feb 24, 2012 at 18:44
• @whuber Thank you for your welcome I am very excited to be part of this community! :-). Conceptually my understanding is the following: A random variable, usually denoted 'big something' - X in this case, can have a PDF. A 'small something' is the one outcome - one observation of big X, we call this 'small x'. So therefore, we can ask "What is the probability of X = 4?". Small x = 4 here, and this has meaning, and we can write "P(X=4) = ?". Big X's PDF evaluated at a chosen small x's value, gives us the probability X come out as x. In other words, P(X=x). (contd) Commented Feb 24, 2012 at 19:00
• @whuber I cannot figure out how to place paragraph spacing in the comments and apparently there is a character limit... I will just edit the question at the bottom to answer you better as far as my understanding. Commented Feb 24, 2012 at 19:03
• Thanks; that's how it's intended to be done. Comments are for improving and clarifying questions, not for extended discussions.
– whuber
Commented Feb 24, 2012 at 19:07
• @whuber Ok I completed the edit for you. Commented Feb 24, 2012 at 19:25

Welcome to the world of statistical thinking, which tends to be backards to mathematical thinking and pretty much every other kind as well.

This may help, distinguish between a pdf and a liklihood. In a pdf theta ($\theta$) is a constant and X is a variable. In a liklihood x is an observed value, or set of values (and therefore a constant) and $\theta$ is unknown (a variable to be solved for). While the pdf and the liklihood look identical, and they bothe provide a link between $\theta$ and X/x, their use and concepts are different (possibly even opposite). Here are a couple of examples that might help:

Consider a goat chained to a stake in a field. I place the stake in the position that I want it and I know the length of the chain, the stake is $\theta$ and the chain is the pdf that relates the value of X (the goats position) to the stake ($\theta$). I know where the stake is, but I don't know where exactly the goat will be at any one time in the future, but the chain/pdf limits where the goat can be and tells me something about where it is likely to be.

Now you come to the field at night when it is to dark to see the stake, but you have an inferred or motion sensor that tells you where the goat is (the value of x) at that time and you want to know where the stake is (so you don't trip on it, or so you can move it, or because you are being followed by a vampire). If you know the length of the chain (which is now the likelihood instead of the pdf, even though it is exacly the same chain) you can use that information along with the position of the goat (x) to tell you about where the stake ($\theta$) is likely to be. If the goat moves to a new position while you are not looking (but you remember where it was and see where it is now) then you have even more information about where the stake is likely to be.

Example 2: As a teacher I choose a value of $\lambda$ for a Poisson distribution that I am going to use to generate data for a homework assignment. At this point I know $\lambda$ but I don't know X yet because I have not generated the data yet. I generate the data $x$'s and give that information to the students, but I don't tell them my $\lambda$ and that I used the Poisson pdf (which they will use as the likelihood). I used $\lambda$ as a fixed value and the Poisson pdf to create random values, the students will use the values (now fixed constants) and the Poisson liklihood (same equation) to estimate the $\lambda$ that they don't know. Now replace the classroom with the real world, the teacher with God, and the student with the statistician/scientist.

1) When you've observed the data, and you're varying the parameter ($\theta$ in your case), you're right - it's not a PDF any more. It's called a likelihood function, and you're trying to find the $\theta$ that maximizes it, hence the term "maximum likelihood." It's a function of $\theta$ with $x$ fixed, rather than a PDF, which is a function of $x$ with $\theta$ fixed. The way you write it in mathematical notation is the same, though.

2) However, if you have a PDF of a vector - your "big X" - and the elements of that vector are independent of each other, the PDF of the entire vector is the product of the PDFs of each individual element of the vector. Note that if the vector is random, at least one element of it must be random or the vector isn't, as a vector is just a collection of elements (well, for purposes of this discussion.) The key here is that you haven't observed the elements yet. After you've observed them, the likelihood function is the product of the likelihood functions associated with each element of the vector - which is what you've got in your product formula.

• Thank you jbowman: "However, if you have a PDF of a vector - your "big X" - and the elements of that vector are independent of each other, the PDF of the entire vector is the product of the PDFs of each individual element of the vector." This is precisely the part I dont get... this is a vector of observations - numbers. Lets say I got [3 4 9 -3 92]. What is the PDF of 3? What is the pdf of 92? Its nonsensical to me... The only way to say 'pdf' here is for 3 to be a random variable (a big X1) ...but its not - its a 'small x1' - an observation... :-/ Commented Feb 24, 2012 at 19:37
• At this point, you've observed the elements of the vector, so you're in likelihood-land, not PDF-land. However, perhaps more to the point, I think perhaps you are confusing the big-X = random variable, little-x = observation terminology with the big-X = vector, little-x = element of vector terminology (just guessing, really.) If I have a vector $X$ of length 10, and each element of that vector $\sim N(0,1)$, then the vector is $\sim MVN(0,I)$, a multivariate normal with mean vector 0 and covariance matrix equal to the identity matrix. Both vector and elements have distributions. Commented Feb 24, 2012 at 19:51
• No no, I am only dealing with one-dimensional case here, lets be clear on that, that is, the 'vector of observations' is not ONE multi-dimensional observation, but rather, N one-dimensional observations. Commented Feb 24, 2012 at 19:55
• They are the same thing, mathematically speaking, although not the same conceptually. Commented Feb 24, 2012 at 21:30
• I realize that... let me phrase the question in a different way. (One-dimensional case of course): The quantity on the left had side of the equation in my OP - is this a probability, or is this a PDF? Commented Feb 24, 2012 at 22:47