Return to Question

Given $N<\infty$, $0<q<1$ (arbitrarily close to 1) and $\epsilon>0$ (arbitrarily small) one can construct an example of a one parameter family of distributions $P_\theta$, $\theta\in[0,1)$, on the unit circle (viewed as a group, namely the unit interval with addition modulo 1) such that $$P_\theta=\theta+P_0$$ and such that if a sample of size $n\leq N$ is drawn from any $P_\theta$, then with probability q, the maximum likelyhood estimate $\hat\theta$ of the parameter $\theta$ will be the worst possible distribution in this family for which the data likelihood is still nonzero.

The distribution $P_0$ depends on N,q and $\epsilon$ and is pretty simple but the density has one thin spike and the maximum likelihood estimate is obtained by shifting $P_\theta$ so that the support of the spike contains as many points of the sample as possible while maintaining a nonzero data likelihood. Details will be provided if requested.

Does this bother you? If not, why can we still trust Maximum Likelihood estimation? Where are the conditions spelled out under which it can be trusted?

Some details: $P_0$ is a convex combination of uniform distributions $$ P=\alpha U_{[0,\delta]}+\beta U_{(\delta,1/4)}+\gamma U_{[1/4,1/2]} $$ supported on $[0,1/2]$. The idea is that $\delta$ is chosen very small so that the piece wise constant density has a large spike at the left end of the support of $P_0$.

Given $N<\infty$, $0<q<1$ (arbitrarily close to 1) and $\epsilon>0$ (arbitrarily small) one can construct an example of a one parameter family of distributions $P_\theta$, $\theta\in[0,1)$, on the unit circle (viewed as a group, namely the unit interval with addition modulo 1) such that $$ P_\theta=\theta+P_0 $$ and such that if a sample of size $n\leq N$ is drawn from any $P_\theta$, then with probability $q$, the maximum likelihood estimate $\hat\theta$ of the parameter $\theta$ will be the worst possible distribution in this family for which the data likelihood is still nonzero.

The distribution $P_0$ depends on $N,q$ and $\epsilon$ and is pretty simple but the density has one thin spike and the maximum likelihood estimate is obtained by shifting $P_\theta$ so that the support of the spike contains as many points of the sample as possible while maintaining a nonzero data likelihood. Details will be provided if requested.

Does this bother you? If not, why can we still trust Maximum Likelihood estimation? Where are the conditions spelled out under which it can be trusted?

Some details: $P_0$ is a convex combination of uniform distributions $$ P=\alpha U_{[0,\delta]}+\beta U_{(\delta,1/4)}+\gamma U_{[1/4,1/2]} $$ supported on $[0,1/2]$. The idea is that $\delta$ is chosen very small so that the piece wise constant density has a large spike at the left end of the support of $P_0$.

Easy inequalities show that any $\theta$ such that $\min x_j\in S(\theta)$ maximizes the likelihood (if $\delta$ is chosen small enough so that the spike is large enough).

Thus $\hat\theta=\min x_j$ is a likelihood maximizer but is not the only one. To make $\hat\theta=\min x_j$ uniquely determined simply make the density of $P_0$ on the interval $[0,\delta]$ slightly sloping down (instead of constant).

The density $f_0$ on the interval $[1/4,1/2]$ is bigger than on the interval $(\delta,1/4)$. From this it follows that $\hat\theta=\min x_j-\delta$ maximizes the likelihood (if $\delta$ is chosen small enough so that the spike is large enough), since this $\theta$ shifts the point $\min x_j$ into the support of the spike while maintaining as many points as possible in the interval $\theta+[1/4,1/2]$.

Thus $\hat\theta=\min x_j-\delta$ is a likelihood maximizer but may not be the only one. This gets us close to the claims.

To make $\hat\theta=\min x_j$ the unique maximizer make the density of $P_0$ on the interval $[0,\delta]$ sloping down with $f_0(0)>>f_0(\delta)$ and $f_0(\delta)$ still very large. Then $f_\theta(\theta)>>f_\theta(\theta+\delta)$ so that shifting the point $min x_j$ to the locus of the maximum of the density $f_\theta$ ensures likelihood maximization.

Obviously some detailed calculations are necessary to verify that $\hat\theta=\min x_j$ really does maximize the likelihood.

Then you choose $\delta>0$ very small so that the spike of the density on $[0,\delta]$ is very large. If this is chosen small enough the likelihood under $P_\theta$ will be maximized by shifting $P_0$ to the right so as to shift as many sample points $x_j$ into the support of the spike as possible while making sure all sample points are still in the support of $P_\theta$ (i.e. the data likelihood is still nonzero), in short $$ \hat\theta=\min x_j $$ and this satisfies the claims above. There is some freedom in the choice of $\alpha$ and $\beta$ subject to only $\alpha,\beta>0$ and $\alpha+\beta+\gamma=1$. Then $\delta$ will depend on these.

Obviously some calculations are left out in this description.

Given $\alpha,\beta,\gamma$ we note that the density $f_0$ of $P_0$ satisfies $0<c<f_0<C$ on $(\delta,1/2]$, where the constants $c,C$ are independent of $\delta$, while $f_0\uparrow\infty$ on the support $S=[0,\delta]$ of the spike of the density $f_0$, for all $0<\delta<1/8$ (explicit calculation).

By translation the same inequalities hold true for the density $f_\theta$ of $P_\theta$ on $\theta+(\delta,1/2]$ respectively the support $S(\theta)=\theta+[0,\delta]$ of the spike of the density $f_\theta$.

Now choose $\delta>0$ so small that $S(\theta)$ can contain at most one sample point $x_j$ and the density $f_0$ of $P_0$ on the interval $[0,\delta]$ is very large. For $\theta>\min x_j$, the data likelihood is zero, since the sample point $\min x_j$ is not in the support of $P_\theta$.

Thus any likelihood maximizing $\theta$ must satisfy $\theta\leq\min x_j$ and so $S(\theta)$ contains no sample point or exactly one sample point, namely the point $\min x_j$.

Easy inequalities show that any $\theta$ such that $\min x_j\in S(\theta)$ maximizes the likelihood (if $\delta$ is chosen small enough so that the spike is large enough).

Thus $\hat\theta=\min x_j$ is a likelihood maximizer but is not the only one. To make $\hat\theta=\min x_j$ uniquely determined simply make the density of $P_0$ on the interval $[0,\delta]$ slightly sloping down (instead of constant).