Confusion on Maximum Likelihood's starting setup equation

Question

This is my first post on CV.

I am wrapping my head around Maximum Likelihood Estimation (MLE), and ironically, I understand everything about the algorithm, except the starting point. What confuses me is the following:

All the examples I have seen of course, start with some vector of observations, x1, x[2], ... x[N]. The observations are iid. The objective is to estimate parameters (theta) of the PDF - ONE PDF - from which sired such observations. This part I get. We are allowed to estimate ONE PDF because X ('big X') is a random variable from which all those x's (small x's) are sired.

As far as I know, big X's have PDFs. Small x's dont. Anyway, so the PDF whos theta we want to estimate is written like this:

p(x1, x[2], x[3]... x[n] | theta) = p(x1|theta) * p(x[2]|theta) * ... p(x[N]|theta).

Like I said if I accept this I know the rest of the algorithm, but this starting point is where I dont understand. Here all the sudden I now have N PDFs. This confuses me a lot:

Like I said before, as far as I know, observations (like x[5]) dont have PDFs. Random variables do. So in this case the observations are 'small x', where as a random variable is 'big X'. We are trying to estimate the pdf of 'big X' - X. How is it that I now have an individual PDFfor each and every observation 'small x'?

Every lecture video/article seem to use contradictory terms among themselves of 'random variable' and 'observation' while simultaneously taking the effort to point out that they are not the same and shouldnt be confused - so which one is it??

Thanks in advance!

EDIT for whuber:

For additional context, here is my current state of understanding: By the way, I am going to assume a Gaussian process for X from here on out for this example.

• A random variable, usually denoted 'big something' - X in this case, can have a PDF. A 'small something' is the one outcome - one observation of big X, we call this 'small x'.

• So therefore, we can ask "What is the probability of X = 4?". Small x = 4 here, and this has meaning, and we can write "P(X=4) = ?". Big X's PDF evaluated at a chosen small x's value, gives us the probability X come out as x. In other words, P(X=x).

• To me this means only X can have a PDF. (X represents a name for a process that can generate many possible values - he can take many values), while x, cannot have a PDF. (What is the PDF of the number 7?). This is how I see it.

• If I have one observation, (x = 7), then I understand that the jist of the MLE is to say, which guassian, with what mu and sigma, maximize the probability of this one number being generated from it? If I had all the time in the universe, I would: 1) Take a guassian PDF with some mu and some sigma, and then 2) plug x = 7 into it and get a probability of it being 7. Eventually I would realize that the best mu for my one x = 7 is in fact when mu = 7.

• If I have two observations, (x1 = 7, x2 = 9), then my question in english is, "What possible combo of mu and sigma did my X have, such that I could have gotten x1 = 7 and x2 = 9 with the highest odds?". To me this means it is nonsensical to assume that x1 and x2 were aberrations and on the tail end of a gaussian. So not knowing anything else, we assume that well, if they are here, they were the most likely. So what values of mu and sigma most likely led to those being generated?

• This now means ok, what is the probability of x1=7 AND x2=9? So P(x1 = 7 AND x2 = 9) = P(x1=7) * P(x2=9). We can do this because they are IID.

EDIT 2

I have attached an image of what I believe to me a Likelihood function for ONE observation, (x = 3), assuming it came from a gaussian X, based on my new understanding from the convos here. Interesting...looks like the light pattern out of the headlight of a car...

Answer 1

Welcome to the world of statistical thinking, which tends to be backards to mathematical thinking and pretty much every other kind as well.

This may help, distinguish between a pdf and a liklihood. In a pdf theta ($\theta$) is a constant and X is a variable. In a liklihood x is an observed value, or set of values (and therefore a constant) and $\theta$ is unknown (a variable to be solved for). While the pdf and the liklihood look identical, and they bothe provide a link between $\theta$ and X/x, their use and concepts are different (possibly even opposite). Here are a couple of examples that might help:

Consider a goat chained to a stake in a field. I place the stake in the position that I want it and I know the length of the chain, the stake is $\theta$ and the chain is the pdf that relates the value of X (the goats position) to the stake ($\theta$). I know where the stake is, but I don't know where exactly the goat will be at any one time in the future, but the chain/pdf limits where the goat can be and tells me something about where it is likely to be.

Now you come to the field at night when it is to dark to see the stake, but you have an inferred or motion sensor that tells you where the goat is (the value of x) at that time and you want to know where the stake is (so you don't trip on it, or so you can move it, or because you are being followed by a vampire). If you know the length of the chain (which is now the likelihood instead of the pdf, even though it is exacly the same chain) you can use that information along with the position of the goat (x) to tell you about where the stake ($\theta$) is likely to be. If the goat moves to a new position while you are not looking (but you remember where it was and see where it is now) then you have even more information about where the stake is likely to be.

Example 2: As a teacher I choose a value of $\lambda$ for a Poisson distribution that I am going to use to generate data for a homework assignment. At this point I know $\lambda$ but I don't know X yet because I have not generated the data yet. I generate the data $x$'s and give that information to the students, but I don't tell them my $\lambda$ and that I used the Poisson pdf (which they will use as the likelihood). I used $\lambda$ as a fixed value and the Poisson pdf to create random values, the students will use the values (now fixed constants) and the Poisson liklihood (same equation) to estimate the $\lambda$ that they don't know. Now replace the classroom with the real world, the teacher with God, and the student with the statistician/scientist.

Answer 2

1) When you've observed the data, and you're varying the parameter ($\theta$ in your case), you're right - it's not a PDF any more. It's called a likelihood function, and you're trying to find the $\theta$ that maximizes it, hence the term "maximum likelihood." It's a function of $\theta$ with $x$ fixed, rather than a PDF, which is a function of $x$ with $\theta$ fixed. The way you write it in mathematical notation is the same, though.

2) However, if you have a PDF of a vector - your "big X" - and the elements of that vector are independent of each other, the PDF of the entire vector is the product of the PDFs of each individual element of the vector. Note that if the vector is random, at least one element of it must be random or the vector isn't, as a vector is just a collection of elements (well, for purposes of this discussion.) The key here is that you haven't observed the elements yet. After you've observed them, the likelihood function is the product of the likelihood functions associated with each element of the vector - which is what you've got in your product formula.