Given $N<\infty$, $0<q<1$ (arbitrarily close to 1) and $\epsilon>0$ (arbitrarily small) one can construct an example of a one parameter family of distributions $P_\theta$, $\theta\in[0,1)$, on the unit circle (viewed as a group, namely the unit interval with addition modulo 1) such that $$ P_\theta=\theta+P_0 $$ and such that if a sample of size $n\leq N$ is drawn from any $P_\theta$, then with probability $q$, the maximum likelihood estimate $\hat\theta$ of the parameter $\theta$ will be the worst possible distribution in this family for which the data likelihood is still nonzero.
Here "worst possible" is understood as $|\theta-\hat\theta|$ is maximal (and this probably implies that various distance measures like total variation distance, Kullback-Leibler distance are also maximized although I have not checked this).
In particular it satisfies $P_{\hat\theta}(support(P_\theta))<\epsilon$ and $P_{\theta}(support(P_{\hat\theta}))<\epsilon$.
The distribution $P_0$ depends on $N,q$ and $\epsilon$ and is pretty simple but the density has one thin spike and the maximum likelihood estimate is obtained by shifting $P_\theta$ so that the support of the spike contains as many points of the sample as possible while maintaining a nonzero data likelihood. Details will be provided if requested.
Does this bother you? If not, why can we still trust Maximum Likelihood estimation? Where are the conditions spelled out under which it can be trusted?
Some details: $P_0$ is a convex combination of uniform distributions $$ P=\alpha U_{[0,\delta]}+\beta U_{(\delta,1/4)}+\gamma U_{[1/4,1/2]} $$ supported on $[0,1/2]$. The idea is that $\delta$ is chosen very small so that the piece wise constant density has a large spike at the left end of the support of $P_0$.
By translation invariance of the family $P_\theta$ we may assume that we are sampling from the probability $P_0$. The claimed properties of the maximum likelihood estimate will be verified for all samples $X=(x_j)\subseteq[1/4,1/2]$.
The parameters $\alpha$, $\beta$, $\gamma$ and $\delta$ are chosen depending on $N,q,\epsilon$. Note that a random draw $x$ from $P_0$ will come from $U_{[1/4,1/2]}$, equivalently $x\in[1/4,1/2]$, with probability $\gamma$.
First you choose $\gamma<1$ so close to one that any sample $X$ of size $n\leq N$ satisfies $X\subseteq[1/4,1/2]$ with probability at least $q$, i.e. $\gamma^N\geq q$.
Given $\alpha,\beta,\gamma$ we note that the density $f_0$ of $P_0$ satisfies $0<c<f_0<C$ on $(\delta,1/2]$, where the constants $c,C$ are independent of $\delta$, while $f_0\uparrow\infty$ on the support $S=[0,\delta]$ of the spike of the density $f_0$, for all $0<\delta<1/8$ (explicit calculation).
By translation the same inequalities hold true for the density $f_\theta$ of $P_\theta$ on $\theta+(\delta,1/2]$ respectively the support $S(\theta)=\theta+[0,\delta]$ of the spike of the density $f_\theta$.
Now choose $\delta>0$ so small that $S(\theta)$ can contain at most one sample point $x_j$ and the density $f_0$ of $P_0$ on the interval $[0,\delta]$ is very large. For $\theta>\min x_j$, the data likelihood is zero, since the sample point $\min x_j$ is not in the support of $P_\theta$.
Thus any likelihood maximizing $\theta$ must satisfy $\theta\leq\min x_j$ and so $S(\theta)$ contains no sample point or exactly one sample point, namely the point $\min x_j$.
The density $f_0$ on the interval $[1/4,1/2]$ is bigger than on the interval $(\delta,1/4)$. From this it follows that $\hat\theta=\min x_j-\delta$ maximizes the likelihood (if $\delta$ is chosen small enough so that the spike is large enough), since this $\theta$ shifts the point $\min x_j$ into the support of the spike while maintaining as many points as possible in the interval $\theta+[1/4,1/2]$.
Thus $\hat\theta=\min x_j-\delta$ is a likelihood maximizer but may not be the only one. This gets us close to the claims.
To make $\hat\theta=\min x_j$ the unique maximizer make the density of $P_0$ on the interval $[0,\delta]$ sloping down with $f_0(0)>>f_0(\delta)$ and $f_0(\delta)$ still very large. Then $f_\theta(\theta)>>f_\theta(\theta+\delta)$ so that shifting the point $min x_j$ to the locus of the maximum of the density $f_\theta$ ensures likelihood maximization.
Obviously some detailed calculations are necessary to verify that $\hat\theta=\min x_j$ really does maximize the likelihood.
