I am comparing multiple published equation forms, refit with independent data. I'm trying to be true to the original authors' methods as much as possible. Therefore, I have 3 linear equations (fit in R using lm()), two of which use transformed Y-variables, and one equation fit using nonlinear regression (fit in R using the gnls() function).

In all instances cases I'm weighting the residual variance structure using the inverse of one of the predictors to account for observed heteroskedasticity.

I have been evaluating the models using R2, and RMSE- using back-transformed data for the two models with transformations.

I've calculated RMSE "by hand" using the following equation:


Should I use similar code to calculate RMSE for the linear and nonlinear regression models? Is the metric still a valid statistic for comparison, or am I missing some important assumption?

Edited: I initially stated that I was also comparing models using AIC; I later recalled that AIC would not be appropriate if the Y-variables were transformed because the models would be estimating different things.


2 Answers 2

  • RMSE is certainly appropriate also for nonlinear models
  • However, the RMSE expressions I know actually calculate the mean, so no -2 (looks like d.f. for linear model? - d.f. for nonlinear models would be different!)
  • In general, I'd not use the residuals for calculating RMSE but rather use independent test cases to avoid an optimistic bias.

From Wikipedia,

RMSE is a frequently used measure of the differences between values predicted by a model or an estimator and the values actually observed.


RMSE is a good measure of accuracy, but only to compare forecasting errors of different models for a particular variable and not between variables, as it is scale-dependent.

Clearly, you can use RMSE for comparing the forecast error between different models (linear or nonlinear), as long as you are comparing models for a particular variable. I don't see any assumptions to consider.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.