Can I compare models from linear regression and nonlinear regression using RMSE?

Question

I am comparing multiple published equation forms, refit with independent data. I'm trying to be true to the original authors' methods as much as possible. Therefore, I have 3 linear equations (fit in R using lm()), two of which use transformed Y-variables, and one equation fit using nonlinear regression (fit in R using the gnls() function).

In all instances cases I'm weighting the residual variance structure using the inverse of one of the predictors to account for observed heteroskedasticity.

I have been evaluating the models using R2, and RMSE- using back-transformed data for the two models with transformations.

I've calculated RMSE "by hand" using the following equation:

 RMSE<-sqrt(sum(residuals(Equation)^2)/length(residuals(Equation))-2))

Should I use similar code to calculate RMSE for the linear and nonlinear regression models? Is the metric still a valid statistic for comparison, or am I missing some important assumption?

Edited: I initially stated that I was also comparing models using AIC; I later recalled that AIC would not be appropriate if the Y-variables were transformed because the models would be estimating different things.

Answer 1

• RMSE is certainly appropriate also for nonlinear models
• However, the RMSE expressions I know actually calculate the mean, so no -2 (looks like d.f. for linear model? - d.f. for nonlinear models would be different!)
• In general, I'd not use the residuals for calculating RMSE but rather use independent test cases to avoid an optimistic bias.

Answer 2

From Wikipedia,

RMSE is a frequently used measure of the differences between values predicted by a model or an estimator and the values actually observed.

Further,

RMSE is a good measure of accuracy, but only to compare forecasting errors of different models for a particular variable and not between variables, as it is scale-dependent.

Clearly, you can use RMSE for comparing the forecast error between different models (linear or nonlinear), as long as you are comparing models for a particular variable. I don't see any assumptions to consider.