I have developed a predictive model in R which predicts a numerical value. To, check model accuracy, I have included root mean square error, root mean square error, bias and variance, calculated using functions inherited from the package "Metrics".

I am aware that low values of these statistics indicate a good model.

When its comes to comparing accuracy of two models, which statistic is to be considered for comparison? Is there any other test that should be performed to check accuracy?

I am new to predictive modelling, so I desire to know any other methods or tests to better indicate prediction accuracy and model comparison.

  • 1
    $\begingroup$ No such thing as best, which metric you use depends entirely on your goal. $\endgroup$ – user2974951 Jan 23 '20 at 9:47
  • $\begingroup$ What exact definition would be defined as a goal? My goal is to generate a predicted value that is as close as the actual value. I am using a regression model. $\endgroup$ – tsu Jan 23 '20 at 9:52

You are mixing different points:

  1. Loss function: what you use to train and optimize your model
  2. Evaluation metric: what you use to evaluate your model (predictions)
  3. Test statistic: what you might use to formally compare two models

These are different concepts which have different purposes. From your comments it looks like you might be interested in #2, in which case, if you have no specific goal in mind, you can use MSE for regression.

There are no "best" metrics. For ex., MSE measures the squared deviation from the truth, while MAE measures the absolute deviation from the truth. What is "best" for you only you can decide, based on how you wish to evaluate a model.

If you wish to use #2 only to compare two models then it does not matter that much which metric you choose. If you wish to improve model training (#1) then you want to change the loss function.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.