In maximum likelihood estimation, I sometimes see the average likelihood function $\frac{1}{N}\mathcal{L}(\theta, x)$ used over the likelihood. Since maximum likelihood estimation optimises over the parameters $\theta$, I dont see the motivation for averaging, since it doesnt change the maximim of $\mathcal{L}(\theta, x)$. Can someone give me some intuition for why we might want to divide by $N$?

  • 3
    $\begingroup$ One reason to divide by $N$ would be to make numbers comparable over data sets with different sample sizes. $\endgroup$ – kjetil b halvorsen Mar 16 '17 at 14:17

TLDR version:

If you are using a first order optimization algorithm, such as gradient ascent, using the average likelihood as your objective function stabilizes the behavior of algorithm as the sample size changes. On the other hand, if you are using a second order optimization algorithm, such as Newton's Method, whether you use likelihood or normalized likelihood is irrelevant.

More Details

In many problems, the log likelihood can be written as

$L(\theta|x) = \sum_{i = 1}^n L(\theta|x_i)$

i.e. the log likelihood is the sum of contributions from each observation. In standard gradient ascent, we update $\theta$ with

$\theta^{(t+1)} = \theta^{(t)} + \alpha \frac{\partial L}{\partial \theta^{(t)}}$

Here, $\alpha$ is often referred to as the "learning rate", and in many problems, is selected without strong justification (i.e. empirically seems to work well, but no proof that is optimal). Choosing a poor learning rate can lead to very poor performance.

Now, the tricky thing is that we need to realize that

$\frac{\partial L}{\partial \theta} = \sum_{i = 1}^n \frac{\partial L(\theta | x_i)}{\partial \theta}$

This means that as our sample size gets larger, we take larger steps when using the unnormalized log-likelihood, even when we are the same distance away from the optimal solution. With that in mind, we can see that it should be impossible to pick an $\alpha$ that is optimal for all sample sizes.

However, if we use the normalized log-likelihood, then our step size should be approximately the same size as the sample size grows. Thus, $\alpha$ is likely to be less sensitive to sample size.

When using Newton's Method, this is not an issue: the step is

$\theta^{(t+1)} = \theta - H^{-1} \frac{\partial L}{\partial \theta^{(t)}}$

where $H$ is the Hessian. Because the Hessian is affected by the multiplication of normalizing constants in the same way, multiplying the objective function by $1/n$ does not change the step size.

  • 1
    $\begingroup$ +1. The issue you raised is important for minibatch training, or situations where new data becomes available. It may not affect much for batch training on static datasets, because the learning rate must be tuned for each problem anyway. The issue isn't related to first vs. second order methods. For example, a first order line search method (e.g. using the gradient to choose the step direction and line search to choose the step size on each iteration) wouldn't have this problem. $\endgroup$ – user20160 Aug 7 '17 at 1:59
  • 1
    $\begingroup$ @user20160: the reason I bring up 1st vs 2nd order is that a lot of us (me included) started doing optimization with 2nd order methods. In this case, multiplying the likelihood by a constant makes no difference at all; the constant in the first derivative cancels with the constant in the second. So we instinctively think this makes no difference, at least in terms of optimization. But it can make a difference in 1st order methods (although, as you note, if you use a line search, it is no longer really an issue). $\endgroup$ – Cliff AB Aug 7 '17 at 3:49

One reason to divide by $N$ would be to make numbers comparable over data sets with different sample sizes. Otherwise it does not seem a very important issue.

  • 1
    $\begingroup$ As in my answer, it can make a difference if you are using something like gradient descent as your optimization algorithm. $\endgroup$ – Cliff AB Aug 7 '17 at 0:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.