Making sense of MLE

Question

Since we were taught MLE (Maximum Likelihood Estimation), a number of questions often bothered me. Why does Maximum Likelihood Estimation work ? Why does it always produce almost accurate results while estimating unknown parameters of some specified distribution ?

---

A serious doubt which I have is about likelihood. Suppose we're a given some data, say $X_1, X_2, \cdots, X_n$ from some distribution , e.g. Poisson or Gamma or Normal (or anything else) with some unknown parameter(s). Now, in order to calculate the MLE for the unknown parameters, we construct the likelihood function at first, which, I guess, is nothing but the probability of choosing that very sample. This logic is completely fine if the samples are coming from a discrete distribution.

But, for continuous distributions, why does this the same method hold ? Then, total probability of choosing that very sample is $0$ of course. So, for continuous distributions, why do we consider the value of the PDF of the specified distribution at the sample points, and multiply them for obtaining the likelihood function ?

Answer 1

With continuous distributions you are right to say that all samples would have probability zero, yet some will have larger density that others. The logic is the same as with discrete distributions, you choose the values of the parameters which would give the observed sample maximum density.

Why does it work so well? Not always does, in fact optimality properties like asymptotic unbiasedness and efficiency hold for large samples, while in small samples the MLE estimator might be suboptimal.