I have a model for predicting a trajectory (x as a function of time) with several parameters. At the moment, I calculate the root mean square error (RMSE) between the predicted trajectory and the experimentally recorded trajectory. Currently, I minimise this difference (the RMSE) using simplex (fminsearch in matlab). While this method works to give good fits, I would like to compare several different models, so I think I need to compute the likelihood so that I can use maximum likelihood estimation rather than minimising the RMSE (and then compare the models using AIC or BIC). Is there any standard way of doing this?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
The root mean squared error and the likelihood are actually closely related. Say you have a dataset of $\lbrace x_i, z_i \rbrace$ pairs and you want to model their relationship using the model $f$. You decide to minimize the quadratic error
$$\sum_i \left(f(x_i) - z_i\right)^2$$
Isn't this choice totally arbitrary? Sure, you want to penalize estimates that are completely wrong more than those that are about right. But there is a very good reason to use the squared error.
Remember the Gaussian density: $\frac{1}{Z}\exp \frac{-(x - \mu)^2}{2\sigma^2}$ where $Z$ is the normalization constant that we do not care about for now. Let's asume that your target data $z$ is distributed according to a Gaussian. So we can write down the likelihood of the data.
$$\mathcal{L} = \prod_i \frac{1}{Z}\exp \frac{-(f(x_i) - z_i)^2}{2\sigma^2}$$
Now if you take the logarithm of this...
$$\log \mathcal{L} = \sum_i \frac{-(f(x_i) - z_i)^2}{2\sigma^2} - \log Z$$
... it turns out that it is very closely related to the rms: the only differences are some constant terms, a square root and a multiplication.
Long story short: Minimizing the root mean squared error is equivalent to maximizing the log likelihood of the data.
-
$\begingroup$ Thanks for the clear explanation. So if I want to compare two (non-embedded) models using BIC, I can just drop the sigma^2 and Z terms (effectively assuming they are same across models) when calculating the likelihood? $\endgroup$– JasonOct 5, 2011 at 10:59
-
$\begingroup$ Yes. Both terms depend only on $\sigma$, thus you can drop them if both $\sigma$s are equal. $\endgroup$– bayerjOct 5, 2011 at 11:43
-
2$\begingroup$ I think there is a mistake in the last step above (taking the log of the likelihood), it should be: $$ \log \mathcal{L} = \sum_i \frac{(f(x_i) - z_i)^2}{2\sigma^2} - \log Z $$ This doesn't change the "bottom line" because the log likelihood is linearly related to the RMSE, so minimizing RMSE is equivalent to minimizing log likelihood $\endgroup$– JasonJan 28, 2012 at 11:48
-
2$\begingroup$ Is there a negative sign missing in the Gaussian distribution? $\endgroup$– ManojOct 7, 2014 at 12:24
-
1$\begingroup$ Shouldn't the conclusion be the opposite? Minimizing the sum of squared errors maximizes the log-likelihood (for a fixed $\sigma$), and thus maximizes the likelihood (since log is monotonic). $\endgroup$ Dec 29, 2016 at 10:06