Maximum likelihood and Minimizing Kullback–Leibler divergence to the ecdf?! (i.e.: finite sample statement?)

Question

There is a known relationship stating that finding MLE is asymptotically the same is minimizing Kullback–Leibler divergence (see wiki here), or just the cross entropy. I'm wondering if there is a similar relationship in terms of a finite sample statement. Specifically, I'm wondering if it can be shown that MLE is the same as finding theta hat that minimizes the probability P with it, with the empirical CDF (wiki) using the data. That is, if we'll check the kl divergence between the ecdf of the data, and the estimated cdf of each observation based on the estimated parameter theta, then minimizing this distance will give us the MLE. I have a feeling that this can be proven using the alternative formulation of the expectancy in terms of the CDF (see wiki).

I couldn't find this in the literature from my searches, I'd be happy if someone knows how to prove this and can share (or share a link to a reference). Thanks!

Answer 1

I think that this can't be true. The ECDF/CDF doesn't behave like a density as required by the KL-divergence.

Let's consider the problem of estimating a Gaussian mean with 2 observations, say 0 and 1. Let's say the variance is known to be $\sigma^2=1$. The ML-estimator is 1/2. However the ECDF $F_2$ has $F_2(0)=\frac{1}{2}$ and $F_2(1)=1$. This is fit better by a Gaussian distribution with mean 0, which has $\Phi(0)=\frac{1}{2}$ and $\Phi(1)$ larger (and therefore closer to $F_2(1)$) than the Gaussian with mean 1/2.

Now if we plug this into the formula for the "pseudo KL-divergence" as suggested in the posting (if I understand it correctly), we have, for an estimator $\hat\mu$, $$ F_2(0)*\log\left(\frac{F_2(0)}{\Phi_{\hat\mu,1}(0)}\right)+F_2(1)*\log\left(\frac{F_2(1)}{\Phi_{\hat\mu,1}(1)}\right). $$ As CDFs don't behave like densities or probabilities, actually $\hat\mu\to -\infty$ will minimise this, because it will make both $\Phi_{\hat\mu,1}(0)$ and $\Phi_{\hat\mu,1}(1)$ as big as possible, which wouldn't work for densities (if you make the density big in one place, it will be smaller in another).

So either the "empirical CDF-KL" needs to be something else, or it won't work.

To actually mimic the KL-divergence with CDFs, you'd probably need $P_2(0)$ and $P_2(1)$ (empirical probability values, rather than ECDF values), and then some way to split up the probability mass in the fitting Gaussian according to $\Phi_{\hat\mu,1}(0)$ and $\Phi_{\hat\mu,1}(1)$ so that the resulting probabilities sum to one. Not sure how to do this; chances are it can be defined in some way (for two observations one could just cut in the middle), but even then the best I'd expect is for the statement to hold approximately but not precisely, in general.