15
$\begingroup$

I have a model for predicting a trajectory (x as a function of time) with several parameters. At the moment, I calculate the root mean square error (RMSE) between the predicted trajectory and the experimentally recorded trajectory. Currently, I minimise this difference (the RMSE) using simplex (fminsearch in matlab). While this method works to give good fits, I would like to compare several different models, so I think I need to compute the likelihood so that I can use maximum likelihood estimation rather than minimising the RMSE (and then compare the models using AIC or BIC). Is there any standard way of doing this?

$\endgroup$

1 Answer 1

22
$\begingroup$

The root mean squared error and the likelihood are actually closely related. Say you have a dataset of $\lbrace x_i, z_i \rbrace$ pairs and you want to model their relationship using the model $f$. You decide to minimize the quadratic error

$$\sum_i \left(f(x_i) - z_i\right)^2$$

Isn't this choice totally arbitrary? Sure, you want to penalize estimates that are completely wrong more than those that are about right. But there is a very good reason to use the squared error.

Remember the Gaussian density: $\frac{1}{Z}\exp \frac{-(x - \mu)^2}{2\sigma^2}$ where $Z$ is the normalization constant that we do not care about for now. Let's asume that your target data $z$ is distributed according to a Gaussian. So we can write down the likelihood of the data.

$$\mathcal{L} = \prod_i \frac{1}{Z}\exp \frac{-(f(x_i) - z_i)^2}{2\sigma^2}$$

Now if you take the logarithm of this...

$$\log \mathcal{L} = \sum_i \frac{-(f(x_i) - z_i)^2}{2\sigma^2} - \log Z$$

... it turns out that it is very closely related to the rms: the only differences are some constant terms, a square root and a multiplication.

Long story short: Minimizing the root mean squared error is equivalent to maximizing the log likelihood of the data.

$\endgroup$
6
  • $\begingroup$ Thanks for the clear explanation. So if I want to compare two (non-embedded) models using BIC, I can just drop the sigma^2 and Z terms (effectively assuming they are same across models) when calculating the likelihood? $\endgroup$
    – Jason
    Commented Oct 5, 2011 at 10:59
  • $\begingroup$ Yes. Both terms depend only on $\sigma$, thus you can drop them if both $\sigma$s are equal. $\endgroup$
    – bayerj
    Commented Oct 5, 2011 at 11:43
  • 2
    $\begingroup$ I think there is a mistake in the last step above (taking the log of the likelihood), it should be: $$ \log \mathcal{L} = \sum_i \frac{(f(x_i) - z_i)^2}{2\sigma^2} - \log Z $$ This doesn't change the "bottom line" because the log likelihood is linearly related to the RMSE, so minimizing RMSE is equivalent to minimizing log likelihood $\endgroup$
    – Jason
    Commented Jan 28, 2012 at 11:48
  • 2
    $\begingroup$ Is there a negative sign missing in the Gaussian distribution? $\endgroup$
    – Manoj
    Commented Oct 7, 2014 at 12:24
  • 1
    $\begingroup$ Shouldn't the conclusion be the opposite? Minimizing the sum of squared errors maximizes the log-likelihood (for a fixed $\sigma$), and thus maximizes the likelihood (since log is monotonic). $\endgroup$ Commented Dec 29, 2016 at 10:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.