Measuring error from regression results, is MSE appropriate here?

Question

I used boosted regression trees on a dataset I was working on to predict how much a customer will spend in a given year. Here is a sample of the output:

  Person Actual Predicted
    1     500   400 
    2   300     100 
    3   2000    1900    
    4   5         0 
    5   100000  60000   

I would like to assess the accuracy of the model. However, if I compute the square root of MSE, I get 8981.54, which is too large for customers that will not spend anywhere close to this value. I then tried to compute a correlation coefficient between the two columns, and I get 0.999, which I'm not sure if it is a possible alternative? Any suggestions here?

Thanks!

Answer 1

I take that you have more than 5 customers. Then make a histogram of the $Predicted-Actual$.

If it is symmetric and do not have (a lot of) outliers then the MSE should be OK.

If it has (a lot of) (or large) outliers, then use the MAE.

If it is asymmetric, make a histogram of $\frac {Predicted - Actual} {Actual}$, if this one is symmetric then use the MAPE.

If none of the above can be used, then you are bound to analyse further. I remember a case in which all new products were over-estimated and old product under-estimated. Start by looking at large forecast errors.

Answer 2

Ideally, your error measure should be informed by the loss you will incur because of the action you take based on a forecast. What will you do if you predict 0 for one customer, or 60,000 for a different one? Compare loss-functions, or what is called the Cost of Forecast Error.

In an imperfect world, forecasters use different error measures. In your case, the mape may be informative. Be aware, though, that it has drawbacks: minimizing the MAPE may lead you to systematically biased forecasts.

$R^2$ is very rarely used, because it is pretty uninformative. If your forecast is always half the actual, your $R^2$ will be 1. Same if your forecast is always twice the actual. $R^2$ only measures correlation, and whether a forecast is correlated with the actual is not really all that interesting.