Model performance plotting: MAE plot for regression model

Question

I have a regression model for determining rentals, and the prediction is the number of rentals by hour, not a 0/1 binary outcome.

Therefore for a performance metric I am principally looking at the Mean Absolute Error (MAE).

My question is, is there a way to graphically plot the MAE?

My dataset also needs to look at how the model's performance varies the further it predicts into the future, so I'd like to plot the MAE of the regression prediction on the test data against the time it is predicting.

Ideally I'm hoping for a package in R for plotting the MAE graphically, essentially an MAE equivalent of the ROCR package for plotting the ROC curve?

Answer 1

Of course you can plot lots of MAEs.

• If you have a single time series you are predicting, you can simply plot a time series of your absolute error over the forecasting horizon. (Not much "mean" here.)

• If you have multiple time series, e.g., predictions of rentals at different locations or of different products or car models, then you can average AEs over time series per future time bucket and then plot time series of MAEs.

  If so, I usually find it very enlightening to also plot the time series of quantiles of errors - sometimes the MAEs look fine, but the high quantiles (the "bad boys") are the ones you learn a lot from.

Here is an example:

horizon <- 10
nn <- 20

set.seed(1)
aes <- ts(matrix(exp(rnorm(horizon*nn,log(matrix(1:horizon,nrow=horizon,ncol=nn)+1),1)),
    nrow=horizon,ncol=nn))

foo <- ts(cbind(rowMeans(aes),t(apply(aes,1,quantile,c(0.1,0.9)))))
plot(foo[,1],ylim=range(foo),lwd=2,ylab="",main="Absolute Errors\n10%, Mean, 90%")
lines(foo[,2]); lines(foo[,3])

I am not aware of any R package to do this automatically, but as you see, it's not hard to munge and plot your errors by hand.