19
$\begingroup$

I am evaluating a physical model and would like to know which one of the methods I should be using here (between RMSE and Coefficient of Determination R2)

The problem is as follows: I have a function that outputs predictions for the input value x, $\overline{y_x}= f(x)$. I also have the actual observation for that value that I call $y_x$.

My question is what are the pros and cons of either RMSE or $R^2$. I have seen both of them being used in papers for the problem that I am working on.

$\endgroup$
17
$\begingroup$

I have used them both, and have a few points to make.

  • Rmse is useful because it is simple to explain. Everybody knows what it is.
  • Rmse does not show relative values. If $rmse=0.2$, you must specifically know the range $\alpha <y_x< \beta$. If $\alpha=1, \beta=1000$, then 0.2 is a good value. If $\alpha=0, \beta=1$, it does not seem not so good anymore.
  • Inline with the previous approach, rmse is a good way to hide the fact that the people you surveyed, or the measurements you took are mostly uniform (everybody rated the product with 3 stars), and your results look good because data helped you. If data was a bit random, you would find your model orbiting Jupiter.
  • Use adjusted coefficient of determination, rather than the ordinary $R^2$
  • Coefficient of determination is difficult to explain. Even people from the field needs a footnote tip like \footnote{The adjusted coefficient of determination is the proportion of variability in a data set that can be explained by the statistical model. This value shows how well future outcomes can be predicted by the model. $R^2$ can take 0 as minimum, and 1 as maximum.}
  • Coefficient of determination is however very precise in telling how well your model explains a phenomena. if $R^2=0.2$, regardless of $y_x$ values, your model is bad. I believe cut off point for a good model starts from 0.6, and if you have something around 0.7-0.8, your model is a very good one.
  • To recap, $R^2=0.7$ says that, with your model, you can explain 70% of what is going on in the real data. The rest, 30%, is something you do not know and you cannot explain. It is probably because there are confounding factors, or you made some mistakes in constructing the model.
  • In computer science, almost everybody uses rmse. Social sciences use $R^2$ more often.
  • If you do not need to justify the parameters in your model, just use rmse. However, if you need to put in, remove or change your parameters while building your model, you need to use $R^2$ to show that these parameters can explain the data best.
  • If you will use $R^2$, code in the R language. It has libraries, and you just give it the data to have all results.

For an aspiring computer scientist, it was thrilling to write about statistics . Yours truly.

$\endgroup$
  • 6
    $\begingroup$ This value shows how well future outcomes can be predicted by the model - this is extremely misleading and leaning towards just plain wrong. There's no guarantee that a high coefficient of determination in a given model relates to how well future outcomes will be predicted. $\endgroup$ – Prophet60091 Oct 2 '14 at 16:01
  • 4
    $\begingroup$ I think statements like "if $R^2 = 0.2$ your model is bad", "$R^2 = 0.7-0.8$ your model is very good" are gross generalizations. If anything for a real world problem an $R^2$ of 0.8 would make strongly suspect overfitting problems... $\endgroup$ – usεr11852 Dec 5 '15 at 23:12
  • 2
    $\begingroup$ if $R^2$=0.2, regardless of yx values, your model is bad. I believe cut off point for a good model starts from 0.6, and if you have something around 0.7-0.8, your model is a very good one. This heavily depends on the field you are working in. Imagine that you try to predict stack exchange relevant indices for next year. You would be the richest man in the world with an $R^2$ of 0.2. $\endgroup$ – Jan Hackenberg Sep 2 '16 at 12:59
  • $\begingroup$ I agree with Jan Hackenberg and Prophet60091. Parts of your answer are definetely wrong and I do not understand why this is the accepted answer and people are upvoting. Actually this probably means that people are using their metrics without knowing how to interpret them.. $\endgroup$ – Cord Kaldemeyer Jul 30 '18 at 11:49
8
$\begingroup$

No matter what Errror measurement you give, consider giving your complete result vector in an appendix. People who like to compare against your method but prefer another error measurement can derive such value from your table.

$R^2$:

  • Does not reflect systematic errors. Imagine you measure diameters instead of radii of circular objects. You have an expected overestimation of 100 %, but can reach still an $R^2$ close to 1.

  • Disagree with previous comments that $R^2$ is difficult to understand. The higher the value is, the more precise is your model, but it can include systematical errors.

  • Can be expressed by the easy to understand formula where you build the ratio of sum of squared residuals and divide by the mean:

$R^2 = 1 - {\frac{SS_E}{mean}} = 1 - \frac{\sum{(y_i - \overline{y_i}})^2}{\sum{(y_i - \overline{y}})^2}$

  • should be expressed in its more advanced version of $R^2_{adj.}$. Here more predictors do punish the model. Expected to be more robust against overfitting.

$RMSE$:

  • You can reach a low $RMSE$ only by having both a high precision (single but large outliers punish heavily) and no systematic error. So in a way a low $RMSE$ garantues better quality than a high $R^2$ does.

  • This number has a unit and is for people not familiar with your data not easy to interpret. It can be for example devided with the mean of the data to produce a $rel. RMSE$. Be careful, this is not the only definition of $rel. RMSE$. Some people prefer to divide by the range of their data instead of dividing by the mean.

As other people mentioned, the choice might be dependent on your field and state of the art. Is there a hugely accepted method to compare too? Use the same measurement as they do and you are able to directly link your methods benefits easily in the discussion.

$\endgroup$
5
$\begingroup$

Both the Root-Mean-Square-Error (RMSE) and coefficient of determination ($R^2$) offer different, yet complementary, information that should be assessed when evaluating your physical model. Neither is "better", but some reports might focus more on one metric depending on the particular application.

I would use the following as a very general guide to understanding the difference between both metrics:

The RMSE gives you a sense of how close (or far) your predicted values are from the actual data you are attempting to model. This is useful in a variety of applications where you wish to understand the accuracy and precision of your model's predictions (e.g., modelling tree height).

Pros

  1. It is relatively easy to understand and communicate since reported values are in the same units as the dependent variable being modelled.

Cons

  1. It is sensitive to large errors (penalizes large prediction errors more than smaller prediction errors).

The coefficient of determination ($R^2$) is useful when you are attempting to understand how well your selected independent variable(s) explain the variability in your dependent variable(s). This is useful when you are attempting to explain what factors might be driving the underlying process of interest (e.g., climatic variables and soil conditions related to tree height).

Pros

  1. Gives an overall sense of how well your selected variables fit the data.

Cons

  1. As more independent variables are added to your model, $R^2$ increases (see adj. $R^2$ or Akaike's Information Criterion as potential alternatives).

Of course, the above will be subject to sample size and sampling design, and a general understanding that correlation does not imply causation.

$\endgroup$
1
$\begingroup$

There is also MAE, Mean Absolute Error. Unlike RMSE, it isn't overly sensitive to large errors. From what I've read, some fields prefer RMSE, others MAE. I like to use both.

$\endgroup$
0
$\begingroup$

Actually,for statistical scientists should know the best fit of the model,then RMSE is very important for those people in his robust research.if RMSE is very close to zero,then the model is best fitted.

The coefficient of determination is good for other scientists like agricultural and other fields. It is a value between 0 and 1. If it is 1, 100% of the values matches to the observed data sets. If it is 0 ,then the data completely heterogeneous. Dr.SK.Khadar Babu,VIT University,Vellore,TamilNadu,India.

$\endgroup$
0
$\begingroup$

If some number is added to each element of one of the vectors, RMSE changes. Same if all elements in one of or both vectors are multiplied by a number. R code follows;

#RMSE vs pearson's correlation
one<-rnorm(100)
two<-one+rnorm(100)

rumis<-(two - one)^2
(RMSE<-sqrt(mean(rumis)))
cor(one,two)

oneA<-one+100

rumis<-(two - oneA)^2
(RMSE<-sqrt(mean(rumis)))
cor(oneA,two)

oneB<-one*10
twoB<-two*10

rumis<-(twoB - oneB)^2
(RMSE<-sqrt(mean(rumis)))
cor(oneB,twoB)
cor(oneB,twoB)^2
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.