This is my first post on CV.
I am wrapping my head around Maximum Likelihood Estimation (MLE), and ironically, I understand everything about the algorithm, except the starting point. What confuses me is the following:
All the examples I have seen of course, start with some vector of observations, x1, x, ... x[N]. The observations are iid. The objective is to estimate parameters (theta) of the PDF - ONE PDF - from which sired such observations. This part I get. We are allowed to estimate ONE PDF because X ('big X') is a random variable from which all those x's (small x's) are sired.
As far as I know, big X's have PDFs. Small x's dont. Anyway, so the PDF whos theta we want to estimate is written like this:
Like I said if I accept this I know the rest of the algorithm, but this starting point is where I dont understand. Here all the sudden I now have N PDFs. This confuses me a lot:
Like I said before, as far as I know, observations (like x) dont have PDFs. Random variables do. So in this case the observations are 'small x', where as a random variable is 'big X'. We are trying to estimate the pdf of 'big X' - X. How is it that I now have an individual PDFfor each and every observation 'small x'?
Every lecture video/article seem to use contradictory terms among themselves of 'random variable' and 'observation' while simultaneously taking the effort to point out that they are not the same and shouldnt be confused - so which one is it??
Thanks in advance!
EDIT for whuber:
For additional context, here is my current state of understanding: By the way, I am going to assume a Gaussian process for X from here on out for this example.
A random variable, usually denoted 'big something' - X in this case, can have a PDF. A 'small something' is the one outcome - one observation of big X, we call this 'small x'.
So therefore, we can ask "What is the probability of X = 4?". Small x = 4 here, and this has meaning, and we can write "P(X=4) = ?". Big X's PDF evaluated at a chosen small x's value, gives us the probability X come out as x. In other words, P(X=x).
To me this means only X can have a PDF. (X represents a name for a process that can generate many possible values - he can take many values), while x, cannot have a PDF. (What is the PDF of the number 7?). This is how I see it.
If I have one observation, (x = 7), then I understand that the jist of the MLE is to say, which guassian, with what mu and sigma, maximize the probability of this one number being generated from it? If I had all the time in the universe, I would: 1) Take a guassian PDF with some mu and some sigma, and then 2) plug x = 7 into it and get a probability of it being 7. Eventually I would realize that the best mu for my one x = 7 is in fact when mu = 7.
If I have two observations, (x1 = 7, x2 = 9), then my question in english is, "What possible combo of mu and sigma did my X have, such that I could have gotten x1 = 7 and x2 = 9 with the highest odds?". To me this means it is nonsensical to assume that x1 and x2 were aberrations and on the tail end of a gaussian. So not knowing anything else, we assume that well, if they are here, they were the most likely. So what values of mu and sigma most likely led to those being generated?
This now means ok, what is the probability of x1=7 AND x2=9? So P(x1 = 7 AND x2 = 9) = P(x1=7) * P(x2=9). We can do this because they are IID.
I have attached an image of what I believe to me a Likelihood function for ONE observation, (x = 3), assuming it came from a gaussian X, based on my new understanding from the convos here. Interesting...looks like the light pattern out of the headlight of a car...