Which performance measure to use when using SVM: MSE or MAE?

Question

It is a common practice to measure an SVM model's performance by calculating its MSE (Mean Square Error). Why not use Mean Absolute Error (averaging errors' absolute values instead of squared values)?

Answer 1

Actually, looking at both MAE and RMSE gives you additional information about the distribution of the errors:

$\mathrm{MAE} \leq \mathrm{RMSE} \leq \mathrm{MAE}^2$ (for regression)

• if $\mathrm{RMSE}$ is close to $\mathrm{MAE}$, the model makes many relatively small errors
• if $\mathrm{RMSE}$ is close to $\mathrm{MAE}^2$, the model makes few but large errors

Answer 2

MAE is more intuitive than MSE to simply evaluate the overall error.

MSE is easier to handle mathematically for variance analysis. For example, MSE is used to calculate the error variance $s_e^2$, which is a recurring value in regression statistics.

Answer 3

The choice of performance metric depends on what is important for the application that you are interested in. The MSE is a good performance metric for many applications as there is good reason to suppose that noise process is Gaussian. Sometimes it is better to use the MAE if you don't want your performance metric to be overly sensitive to outliers. Essentially there is no correct performance metric without knowing more about the nature of the application.

On a different note I am not overly keen on support vector regression, as quite often there is a knowledge about the distribution of the noise in the response variable and we are likely to get a better model of the data if we build that expert knowledge into out model. That is why we have GLMs rather than just using least squares regression for everything. The loss function used in SVM does not have a very clear statistical interpretation of this nature. SVM regression also uses a loss function that is based on a sort of worst-case bound on the error, so if you use a performance metric that is essentially an average case statistic, then that suggests you should instead use a model based on average case performance rather than worst case (e.g. a GLM).