6
$\begingroup$

This question really boils down to the following: Under what conditions can we refer to pointwise values of a probability density function? Obviously continuity of the pdf suffices, but because of the measure-theoretic definition of a function, I am at a loss for how to think of the maximum outside of this nice realm.

Consider Maximum Likelihood Estimation. We have a sample of i.i.d. observations, coming from a distribution with an unknown probability density function $f_0({}·{})$. We know that $f_0$ belongs to a certain family of distributions $f({}\cdot{}\mid θ),$ with parameter $θ ∈ Θ $. The the MLE is defined as the argmax of the likelihood function $$L(\theta; x_1,\ldots,x_n) = \prod_{i=1}^n f(\theta, x_i).$$

But in this definition, the MLE depends on the pointwise values of the family of distributions $f({}\cdot{}, \theta)$. Of course, these can be redefined on a set of measure zero. Hence we could redefine the family on our observation point (if our probability space is uncountable) and drastically change the MLE estimate. The random variables defined by this family would be equal almost surely, so that none of their probabilistic properties have been altered.

So is it true that we can really only define the MLE in the discrete or continuous cases? I'm thinking myself in circles. Can somebody clear up this point for me? I also really appreciate references if you have them.

$\endgroup$
1
$\begingroup$

That definition of the likelihood function is an approximation. Typically, you have $n$ observations, which are subject to a measurement error. So, in practice, instead of observing $x_j$, you observe $x_j\pm\epsilon$, where $\epsilon\gt 0$ is the measurement error. So, if you have a sample of $n$ i.i.d. observations, the likelihood function, for a parametric probability model ${\mathbb P}(;\theta)$, is written as: $$L(\theta;x_1,\dots,x_n) = \prod_{j=1}^n {\mathbb P}[(x_j-\epsilon,x_j+\epsilon);\theta].$$

The maximum likelihood estimator is then, the value of the parameter that maximizes the probability under the sample of interest.

Suppose that $F(;\theta)$ is the distribution associated to the probabilities ${\mathbb P}(;\theta)$, then, you get that

$$L(\theta;x_1,\dots,x_n) = \prod_{j=1}^n \{F[x_j+\epsilon;\theta]-F[x_j-\epsilon;\theta]\}.$$ If $\epsilon$ is small enough, by the mean value theorem you have that

$$F[x_j+\epsilon;\theta]-F[x_j-\epsilon;\theta]\approx 2\epsilon f(x_j),$$

Consequently, you obtain the continuous approximation to the likelihood function

$$L(\theta;x_1,\dots,x_n) \propto \prod_{j=1}^n f(x_j;\theta).$$

This relationship is explained in:

Probability and Statistical Inference II: 002 (Universitext) Paperback – 7 Jan 1980 by J.G. Kalbfleisch

$\endgroup$
  • 3
    $\begingroup$ This is interesting, but it appears unsatisfactory for several reasons. One is that your formulation does not correctly model the probability distribution of measurement error. Another is that measurement error is not equivalent to having interval-valued data. A third is that it suggests data with larger errors ought to get more weight in the likelihood, whereas the opposite is true. $\endgroup$ – whuber May 11 '16 at 16:23
0
$\begingroup$

You need the Radon–Nikodym theorem..

In the measure-theoretic problem, you are estimating a distribution $F_0$ that is defined on the sigma-algebra on the sample set, not on the sample set itself. Now suppose you have a parametric model $F(\cdot\mid\theta)$. If this model is dominated by a $\sigma$-finite measure $\mu$, then (by Radon–Nikodym) we can safely move to the associated family of density functions $f(\cdot \mid \theta)$, satisfying $F(A\mid\theta) = \int_A f(y\mid\theta) \, d\mu(y)$.

Clearly, if your measurement lies in a null set for every distribution in your model, your model or your observation is wrong (since the probability of your observation is zero in all distribution that are considered). On the other hand, if there is a distribution for which your measurement does not lie in any null set, then it also can't lie in a null set of $\mu$, and so you are not free to redefine your densities at your observation.

Note, however, that existence nor uniqueness of the MLE are guaranteed.

$\endgroup$
  • $\begingroup$ Why would the Radon–Nikodym theorem be used when it is given at the outset that density functions exist? The Radon–Nikodym theorem merely says that under certain assumptions a density function exists, but that is already among the hypotheses. $\endgroup$ – Michael Hardy Oct 17 at 6:34
  • $\begingroup$ @Michael That assumption doesn't seem to be in evidence: the question concerns cases that are neither discrete nor continuous. $\endgroup$ – whuber Oct 17 at 11:50
  • $\begingroup$ @whuber: It says "But in this definition, the MLE depends on the pointwise values of the family of distributions $f({}\cdot{}, \theta)$." It appears that the poster means that for each $\theta,$ the function $f({}\cdot{}\mid \theta)$ is a density function. $\qquad$ $\endgroup$ – Michael Hardy Oct 18 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.