I want to compare the performance of various volatility models like GARCH, eGARCH, and gjrGARCH from actual volatility (computed using high frequency data). I found three common performance evaluation measures used profoundly in literature, i.e. Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and Mean Absolute Percentage Error (MAPE). But I found conflicting results from the above three criteria, like I found RMSE is minimum for GARCH model, MAE is minimum for eGARCH model, and MAPE is minimum for gjrGARCH model.

I want to know which one is the best measure for evaluating performance and which results should I report? If all three are equally valid then how can I interpret the results? Is there any other way to compare the performance of various volatility models other than performance measures?


What is best depends on your objective or loss function.

For example, if you incur loss that is directly proportional to the absolute deviation of the forecast from the realization, the relevant measure would be mean absolute error (MAE). But perhaps you do not care as much about overpredicting as underpredicting? That would imply an asymmetric loss function. Or perhaps you do not care as long as the loss is smaller than some threshold value? Then you would use a threshold loss function.

In conclusion, which measure to use and to report will depend on your objective or loss function. Just make sure you are assessing out-of-sample performance as in-sample-performance may be superficially good due to overfitting.

  • $\begingroup$ thanks @Richard Hardy. Yes, I am assessing performance on out of sample not in-sample. Can you please suggest some research paper regarding loss function ? $\endgroup$ – Neeraj Nov 30 '15 at 6:16
  • 1
    $\begingroup$ I do not have any good reference right now, so I would google it, and then it is as effective for you to google it yourself. But you could consider the particular application you have from the subject-matter perspective and think what loss could be appropriate. $\endgroup$ – Richard Hardy Nov 30 '15 at 7:15

Not the answer you're looking for? Browse other questions tagged or ask your own question.