up vote 11 down vote favorite
9

I would like to gain a conceptual understanding of Root Mean Squared Error (RMSE) and Mean Bias Deviation (MBD). Having calculated these measures for my own comparisons of data, I've often been perplexed to find that the RMSE is high (for example, 100 kg), whereas the MBD is low (for example, less than 1%).

More specifically, I am looking for a reference (not online) that lists and discusses the mathematics of these measures. What is the normally accepted way to calculate these two measures, and how should I report them in a journal article paper?

It would be really helpful in the context of this post to have a "toy" dataset that can be used to describe the calculation of these two measures.

For example, suppose that I am to find the mass (in kg) of 200 widgets produced by an assembly line. I also have a mathematical model that will attempt to predict the mass of these widgets. The model doesn't have to be empirical, and it can be physically-based. I compute the RMSE and the MBD between the actual measurements and the model, finding that the RMSE is 100 kg and the MBD is 1%. What does this mean conceptually, and how would I interpret this result?

Now suppose that I find from the outcome of this experiment that the RMSE is 10 kg, and the MBD is 80%. What does this mean, and what can I say about this experiment?

What is the meaning of these measures, and what do the two of them (taken together) imply? What additional information does the MBD give when considered with the RMSE?

share|cite|improve this question
  • 2
    Have you looked around our site, Nicholas? Consider starting at stats.stackexchange.com/a/17545 and then explore some of the tags I have added to your question. – whuber♦ May 29 '12 at 13:48
  • @whuber: Thanks whuber!. I've looked around the site, but to me I am still finding it a bit challenging to understand what is really meant in the context of my own research. – Nicholas Kinar May 29 '12 at 15:19
up vote 16 down vote accepted

I think these concepts are easy to explain. So I would rather just describe it here. I am sure many elementary statistics books cover this including my book "The Essentials of Biostatistics for Physicians, Nurses and Clinicians."

Think of a target with a bulls-eye in the middle. The mean square error represent the average squared distance from an arrow shot on the target and the center. Now if your arrows scatter evenly arround the center then the shooter has no aiming bias and the mean square error is the same as the variance.

But in general the arrows can scatter around a point away from the target. The average squared distance of the arrows from the center of the arrows is the variance. This center could be looked at as the shooters aim point. The distance from this shooters center or aimpoint to the center of the target is the absolute value of the bias.

Thinking of a right triangle where the square of the hypotenuse is the sum of the sqaures of the two sides. So a squared distance from the arrow to the target is the square of the distance from the arrow to the aim point and the square of the distance between the center of the target and the aimpoint. Averaging all these square distances gives the mean square error as the sum of the bias squared and the variance.

share|cite|improve this answer
  • Thank you; this is very much appreciated. I am still finding it a little bit challenging to understand what is the difference between RMSE and MBD. As I understand it, RMSE quantifies how close a model is to experimental data, but what is the role of MBD? Maybe my misunderstanding is just associated with terminology. – Nicholas Kinar May 29 '12 at 15:16
  • 1
    The mean bias deviation as you call it is the bias term I described. It measures how far the aimpoint is away from the target. Bias contributes to making the shot inaccurate. – Michael Chernick May 29 '12 at 15:21
  • Thanks again, Michael. So a high RMSE and a low MBD implies that it is a good model? – Nicholas Kinar May 29 '12 at 15:32
  • No a high RMSE and a low MBD just says that the model is poor because of a large variance rather than a large bias. The RMSE is the number that decides how good the model is. – Michael Chernick May 29 '12 at 15:45
  • 1
    @bbadyalina: they are independent pieces of information, in the same way that up/down and left/right are independent. Your question is like asking "if a point is vertically centred, and way off to the left, is it in the middle?", or "If a point is up high, but horizontally in the centre, is it in the middle?" – naught101 Jan 5 at 2:34
up vote 0 down vote

As far I can understand, a RMSE give a more accurate value of the error between model and observed, however the BIAS, in addition to give a value of the error (less accurate than the RMSE), it can also determine if the model is positive bias or negative bias, if the model is underestimating or overestimating the observed values.

share|cite|improve this answer
  • No. You can think of RMSE as the "precision" of a model - e.g. how much spread exists in the errors of it's predictions (note: precision is the inverse of variance - high variance = low precision). And you can think of the Bias as the systematic error in the model - e.g. the average value of all of the errors. The work "accuracy" is a vague combination of both of those, and hence causes a lot of confusion. – naught101 Jan 5 at 2:37
up vote 0 down vote

RMSE is a way of measuring how good our predictive model is over the actual data, the smaller RMSE the better way of the model behaving, that is if we tested that on a new data set (not on our training set) but then again having an RMSE of 0.37 over a range of 0 to 1, accounts for a lot of errors versus having an RMSE of 0.01 as a better model. BIAS is for overestimating or underestimation.

share|cite|improve this answer
  • Could you please provide more details and a worked out example? The OP is looking for an intuitive explanation of the meaning of an RMSE of, say, 100, against his estimation problem. – Xi'an Mar 11 '15 at 10:01
  • This doesn't seem to deliver much intuition. Can you explain more? – Glen_b♦ Mar 11 '15 at 10:55

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.