Is MLE intrinsically connected to logs?

Question

My mathematical exploration led me the following claim:

Claim: MLE is fundamentally connected to logs (and KL divergence, which also uses logs). It’s not correct to say log shows up simply to make math simpler or numerical computations more stable. There’s an inextricable theoretical connection between logs and MLE.

Is this true? I arrived at this conclusion when I was trying to justify how one might have discovered MLE and proved its consistency in the context of large IID samples:

The real world pulls from random variable $X_{\tilde \theta}$. We don’t know $X_{\tilde \theta}$, but we do assume it belongs to the family of distributions $\{P(\theta) : \theta \in S\}$. Denote their PDFs as $f_\theta(x)$.

Suppose by observing the world, we built up a pretty good estimate of $f_{\tilde\theta}$. We don’t know $\tilde \theta$ though; our estimate of the function is just a black-box.

Perhaps our goal is to identify $\tilde \theta$. This works when $\theta$ is identifiable with respect to $P(\theta)$, i.e. the mapping $\theta \mapsto f_\theta$ is injective. Or perhaps $\theta$ is not necessarily identifiable, but our goal is just to pick the best density $f_\theta$ from our family to describe our data.

Either way, we want to minimize the distance between functions $f_{\tilde \theta}$ and $f_{\theta}$.

How to define distance between functions, though? One way is to weight every $x$ by how often it occurs, i.e., $f_{\tilde \theta}(x)$. Thus, we find the $\theta$ minimizing $\int_{-\infty}^{\infty} f_{\tilde \theta}(x) \times \left[ f_{\tilde\theta}(x) - f_{\theta}(x) \right] = \mathbb{E}\, \left[f_{\tilde \theta}(X_{\tilde \theta}) - f_{\theta}(X_{\tilde \theta}) \right]$.

Since the first term is constant, and the second has a minus in front of it, we can find the $\theta$ that minimizes the above expression by simply maximizing $\mathbb{E}\, f_\theta(X_{\tilde\theta})$. This is the raison d‘être of the ‘true’ likelihood function $Li(\theta) = \mathbb{E}\, f_\theta(X_{\tilde\theta})$.

Of course, in the real world, we don’t have access to $f_{\tilde \theta}$ directly. However, we can observe $\tilde x_1, \ldots, \tilde x_n$. By some law of large numbers, $\text{avg}_i f_\theta(\tilde X_i) \to \mathbb{E}\, f_\theta(X_{\tilde \theta})$ in probability, where $\tilde X_i \stackrel{\text{i.i.d.}}{\sim} P(\tilde \theta)$.

Thus, we can conclude by letting our $\hat\theta = \text{argmax}_\theta\, \sum_i f_\theta(\tilde x_i)$ to find a consistent estimate of $\tilde \theta$.

Uh oh! That’s not MLE. The sum in the above expression should be a product. However, if I go back the “How to define distance between functions” part and use KL divergence instead of my distance function based upon weighting each point by how probable it is, then I get

We should minimize

$$ KL(f_{\tilde \theta}, f_\theta) = \int_{-\infty}^{\infty} f_{\tilde\theta}(x) \left[ \log f_{\tilde\theta}(x) - \log f_\theta(x) \right] \, dx $$

which is equivalent to maximizing $\mathbb{E}\, \log f_\theta(X_{\tilde \theta})$, which by some law of large numbers can be approximate with $\text{avg}\, \log f_\theta(\tilde x_i)$, which of course is equivalent to maximizing $\prod_i f_\theta(\tilde x_i) = f_\theta(\mathbf{\tilde x})$.

The above is in fact MLE. And maximizing the product of the $f_\theta$ of each sample makes a lot more sense than maximizing sum.

1. Have I just found a different estimator that’s not the same as MLE, or have I made some error?
2. If the former, is that estimator any good? What makes KL divergence a better distance function here?
3. Does my claim at the beginning of this question about the intrinsic connection between MLE and log hold water?

Answer 1

The answer to your main question can be yes or no, depending on one's perspective.

First, the maximum likelihood principle can be motivated without any logs. In contrast to your approach, one needs to start with the probability of a sample of size $n$ instead of the $n$ probabilities of $n$ samples. There is a reason that it is not the log likelihood principle - from a certain point of view, the likelihood is more fundamental than its log, and taking the log of a product is simply a mathematical convenience. Or one could say that the iid assumption leads to products, and products lead to logs.

But then there are several theories that generalize the ml principle and in which the ml principle corresponds to the log (or some variant like the KL divergence). Most notably there are different classes of other divergences (Renyi, Bregman, ...) that can be used for inference and that lead to consistent estimators, and there is also information geometry. I don't know whether there is a divergence that corresponds to your proposed additive variant, though.

One point that singles out the likelihood and the KL is the Neyman-Pearson Lemma. Another point is the derivation of the ml principle without loss mentioned at the beginning of my answer.