# Failure of Maximum Likelihood Estimation

Given $N<\infty$, $0<q<1$ (arbitrarily close to 1) and $\epsilon>0$ (arbitrarily small) one can construct an example of a one parameter family of distributions $P_\theta$, $\theta\in[0,1)$, on the unit circle (viewed as a group, namely the unit interval with addition modulo 1) such that $$P_\theta=\theta+P_0$$ and such that if a sample of size $n\leq N$ is drawn from any $P_\theta$, then with probability $q$, the maximum likelihood estimate $\hat\theta$ of the parameter $\theta$ will be the worst possible distribution in this family for which the data likelihood is still nonzero.

Here "worst possible" is understood as $|\theta-\hat\theta|$ is maximal (and this probably implies that various distance measures like total variation distance, Kullback-Leibler distance are also maximized although I have not checked this).

In particular it satisfies $P_{\hat\theta}(support(P_\theta))<\epsilon$ and $P_{\theta}(support(P_{\hat\theta}))<\epsilon$.

The distribution $P_0$ depends on $N,q$ and $\epsilon$ and is pretty simple but the density has one thin spike and the maximum likelihood estimate is obtained by shifting $P_\theta$ so that the support of the spike contains as many points of the sample as possible while maintaining a nonzero data likelihood. Details will be provided if requested.

Does this bother you? If not, why can we still trust Maximum Likelihood estimation? Where are the conditions spelled out under which it can be trusted?

Some details: $P_0$ is a convex combination of uniform distributions $$P=\alpha U_{[0,\delta]}+\beta U_{(\delta,1/4)}+\gamma U_{[1/4,1/2]}$$ supported on $[0,1/2]$. The idea is that $\delta$ is chosen very small so that the piece wise constant density has a large spike at the left end of the support of $P_0$.

By translation invariance of the family $P_\theta$ we may assume that we are sampling from the probability $P_0$. The claimed properties of the maximum likelihood estimate will be verified for all samples $X=(x_j)\subseteq[1/4,1/2]$.

The parameters $\alpha$, $\beta$, $\gamma$ and $\delta$ are chosen depending on $N,q,\epsilon$. Note that a random draw $x$ from $P_0$ will come from $U_{[1/4,1/2]}$, equivalently $x\in[1/4,1/2]$, with probability $\gamma$.

First you choose $\gamma<1$ so close to one that any sample $X$ of size $n\leq N$ satisfies $X\subseteq[1/4,1/2]$ with probability at least $q$, i.e. $\gamma^N\geq q$.

Given $\alpha,\beta,\gamma$ we note that the density $f_0$ of $P_0$ satisfies $0<c<f_0<C$ on $(\delta,1/2]$, where the constants $c,C$ are independent of $\delta$, while $f_0\uparrow\infty$ on the support $S=[0,\delta]$ of the spike of the density $f_0$, for all $0<\delta<1/8$ (explicit calculation).

By translation the same inequalities hold true for the density $f_\theta$ of $P_\theta$ on $\theta+(\delta,1/2]$ respectively the support $S(\theta)=\theta+[0,\delta]$ of the spike of the density $f_\theta$.

Now choose $\delta>0$ so small that $S(\theta)$ can contain at most one sample point $x_j$ and the density $f_0$ of $P_0$ on the interval $[0,\delta]$ is very large. For $\theta>\min x_j$, the data likelihood is zero, since the sample point $\min x_j$ is not in the support of $P_\theta$.

Thus any likelihood maximizing $\theta$ must satisfy $\theta\leq\min x_j$ and so $S(\theta)$ contains no sample point or exactly one sample point, namely the point $\min x_j$.

The density $f_0$ on the interval $[1/4,1/2]$ is bigger than on the interval $(\delta,1/4)$. From this it follows that $\hat\theta=\min x_j-\delta$ maximizes the likelihood (if $\delta$ is chosen small enough so that the spike is large enough), since this $\theta$ shifts the point $\min x_j$ into the support of the spike while maintaining as many points as possible in the interval $\theta+[1/4,1/2]$.

Thus $\hat\theta=\min x_j-\delta$ is a likelihood maximizer but may not be the only one. This gets us close to the claims.

To make $\hat\theta=\min x_j$ the unique maximizer make the density of $P_0$ on the interval $[0,\delta]$ sloping down with $f_0(0)>>f_0(\delta)$ and $f_0(\delta)$ still very large. Then $f_\theta(\theta)>>f_\theta(\theta+\delta)$ so that shifting the point $min x_j$ to the locus of the maximum of the density $f_\theta$ ensures likelihood maximization.

Obviously some detailed calculations are necessary to verify that $\hat\theta=\min x_j$ really does maximize the likelihood.

• While this is an interesting result, it doesn't bother me as ML theory states their results only asysmptotically and one has to expect that in some occasions finite sample performance might be pretty bad. However, can you give a more specific example of such a family of distribution $P_\theta$? Sep 20, 2017 at 15:41
• Yeah, I can't make much of this without knowing how $θ$ relates to the actual distribution. Sep 20, 2017 at 15:43
• Can you give the details? Sep 20, 2017 at 16:43
• @chRrr I have added some details.
– gcc
Sep 20, 2017 at 18:44
• Do you have a reference for this construction,or is it yours? Sep 20, 2017 at 20:09

No, this is not bothersome! The short story: you have explicitly constructed a model that with very high probability will be a bad fit to data generated from the model, so the high spike close to zero in $$P_0$$ is artificial and not representative of the data.

The way this works is that with high probability the very high spike will not contribute any observations, but the spike will dominate contributions to the log likelihood function so that the spike in the estimated distribution will be placed above the minimum value of the data. That will with high probability be the wrong place.

I did some calculations to illustrate this, I will not give all details. I started with choosing $$q=0.99$$, $$N=15$$ (but thinking of simulating a sample with $$n=10$$, say). That gives $$\gamma=0.9994$$ and then distributing the missing probability equally gives $$\alpha=\beta=0.0003$$. Then securing a dominating loglikelihood contribution from the spike, $$\delta=10^{-13}$$ will do. That will give an insanely high spike, in a region with no data. In short, as an approximate representation of the data (with $$n=10$$ observations) this model is nonsense.

A more formal analysis is difficult, but hardly necessary. One reason is that computing the Fisher information for this model is "difficult": the derivative of the loglikelihood (with respect to the parameter $$\theta$$) is mostly zero, with delta function spikes at the borders between the three intervals defining the model. Maybe I come back with some simulations. If you replaced the three uniform parts of the model with three gaussian parts, you would get similar results, but a model easier to analyze.

This is a general phenomenon, it is always possible to construct "models" which will misbehave. So in addition to inference principles like maximum likelihood, we need principles for helping model construction, to get sensible models. Tukey said that models should be "bland" or "hornfree", meaning that the model itself should not import information (or too much of it) into the analysis. The information in formal statistical analysis always come from two sources: the data, and the model. Bayesians are not alone in importing non-sample information into their models! A book-length treatment is this.

It is the statisticians responsibility to make models which do not import too much information into the analysis. Your model decidedly has a very big horn!

• This www2.warwick.ac.uk/fac/sci/statistics/crism/workshops/… is a relevant web presentation which among others contain the Tukey citations. Sep 21, 2017 at 12:23
• @ kjetil b halvorsen A spiky family of densities would occur naturally if you suspect that the data generating distribution has an atom but you don't know where and approximate the atom with a spike in the density. Can we reject this as unnatural?
– gcc
Sep 21, 2017 at 13:40
• Can you propose a natural example? More to the point, approximating the atom with a spike works bad with maxlik methods, it would be better to use a discrete atom! That is done naturally, say, with rainfall modelling and an atom at zero. Using a spike its width and height is largely arbitrary, but makes a huge difference for maxlik. Better thgen use a discrete atom. Sep 21, 2017 at 13:49