I have a 70K x 30 dataset and I want to build a regression model on it. Right now, I am running a bunch of algorithms via Weka tool with cross-validation and I compare the RMSE values reported by Weka in order to decide which model works better.

However, after I experiment with Multi layer perceptron, Linear Regression and a bunch of tree-related algorithms, the best performance I got was K-NN algorithm. Since this algorithm is very naive and instance based, I am not sure if just comparing RMSE is the right way.

When experimenting a Regression model, what kind of process should I follow?


As long as you repeat 10-fold cross-validation many times to achieve adequate precision, RMSE is a good measure for comparison, as is mean absolute error and median absolute error, the latter two being more robust.

  • $\begingroup$ Weka tool also reports mean/median absolute error, so I can compare them. I see correlation coefficient value reported as well. Does it play a role on comparing two regression models? $\endgroup$ – Weka Jun 12 '13 at 4:24
  • $\begingroup$ $R^2$ is another excellent measure, but if calculated the usual way allows for a linear recalibration of the predictions so does not penalize for predictions being off by a constant or a constant multiple. $\endgroup$ – Frank Harrell Jun 12 '13 at 12:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.