Measure-Theoretic Definition of MLE

Question

This question really boils down to the following: Under what conditions can we refer to pointwise values of a probability density function? Obviously continuity of the pdf suffices, but because of the measure-theoretic definition of a function, I am at a loss for how to think of the maximum outside of this nice realm.

Consider Maximum Likelihood Estimation. We have a sample of i.i.d. observations, coming from a distribution with an unknown probability density function $f_0({}·{})$. We know that $f_0$ belongs to a certain family of distributions $f({}\cdot{}\mid θ),$ with parameter $θ ∈ Θ $. The the MLE is defined as the argmax of the likelihood function $$L(\theta; x_1,\ldots,x_n) = \prod_{i=1}^n f(\theta, x_i).$$

But in this definition, the MLE depends on the pointwise values of the family of distributions $f({}\cdot{}, \theta)$. Of course, these can be redefined on a set of measure zero. Hence we could redefine the family on our observation point (if our probability space is uncountable) and drastically change the MLE estimate. The random variables defined by this family would be equal almost surely, so that none of their probabilistic properties have been altered.

So is it true that we can really only define the MLE in the discrete or continuous cases? I'm thinking myself in circles. Can somebody clear up this point for me? I also really appreciate references if you have them.

Answer 1

That definition of the likelihood function is an approximation. Typically, you have $n$ observations, which are subject to a measurement error. So, in practice, instead of observing $x_j$, you observe $x_j\pm\epsilon$, where $\epsilon\gt 0$ is the measurement error. So, if you have a sample of $n$ i.i.d. observations, the likelihood function, for a parametric probability model ${\mathbb P}(;\theta)$, is written as: $$L(\theta;x_1,\dots,x_n) = \prod_{j=1}^n {\mathbb P}[(x_j-\epsilon,x_j+\epsilon);\theta].$$

The maximum likelihood estimator is then, the value of the parameter that maximizes the probability under the sample of interest.

Suppose that $F(;\theta)$ is the distribution associated to the probabilities ${\mathbb P}(;\theta)$, then, you get that

$$L(\theta;x_1,\dots,x_n) = \prod_{j=1}^n \{F[x_j+\epsilon;\theta]-F[x_j-\epsilon;\theta]\}.$$ If $\epsilon$ is small enough, by the mean value theorem you have that

$$F[x_j+\epsilon;\theta]-F[x_j-\epsilon;\theta]\approx 2\epsilon f(x_j),$$

Consequently, you obtain the continuous approximation to the likelihood function

$$L(\theta;x_1,\dots,x_n) \propto \prod_{j=1}^n f(x_j;\theta).$$

This relationship is explained in:

Probability and Statistical Inference II: 002 (Universitext) Paperback – 7 Jan 1980 by J.G. Kalbfleisch

Answer 2

You need the Radon–Nikodym theorem..

In the measure-theoretic problem, you are estimating a distribution $F_0$ that is defined on the sigma-algebra on the sample set, not on the sample set itself. Now suppose you have a parametric model $F(\cdot\mid\theta)$. If this model is dominated by a $\sigma$-finite measure $\mu$, then (by Radon–Nikodym) we can safely move to the associated family of density functions $f(\cdot \mid \theta)$, satisfying $F(A\mid\theta) = \int_A f(y\mid\theta) \, d\mu(y)$.

Clearly, if your measurement lies in a null set for every distribution in your model, your model or your observation is wrong (since the probability of your observation is zero in all distribution that are considered). On the other hand, if there is a distribution for which your measurement does not lie in any null set, then it also can't lie in a null set of $\mu$, and so you are not free to redefine your densities at your observation.

Note, however, that existence nor uniqueness of the MLE are guaranteed.