Best measure of prediction accuracy

Question

I have developed a predictive model in R which predicts a numerical value. To, check model accuracy, I have included root mean square error, root mean square error, bias and variance, calculated using functions inherited from the package "Metrics".

I am aware that low values of these statistics indicate a good model.

When its comes to comparing accuracy of two models, which statistic is to be considered for comparison? Is there any other test that should be performed to check accuracy?

I am new to predictive modelling, so I desire to know any other methods or tests to better indicate prediction accuracy and model comparison.

Answer 1

You are mixing different points:

1. Loss function: what you use to train and optimize your model
2. Evaluation metric: what you use to evaluate your model (predictions)
3. Test statistic: what you might use to formally compare two models

These are different concepts which have different purposes. From your comments it looks like you might be interested in #2, in which case, if you have no specific goal in mind, you can use MSE for regression.

There are no "best" metrics. For ex., MSE measures the squared deviation from the truth, while MAE measures the absolute deviation from the truth. What is "best" for you only you can decide, based on how you wish to evaluate a model.

If you wish to use #2 only to compare two models then it does not matter that much which metric you choose. If you wish to improve model training (#1) then you want to change the loss function.