0
$\begingroup$

I have a regression model for determining rentals, and the prediction is the number of rentals by hour, not a 0/1 binary outcome.

Therefore for a performance metric I am principally looking at the Mean Absolute Error (MAE).

My question is, is there a way to graphically plot the MAE?

My dataset also needs to look at how the model's performance varies the further it predicts into the future, so I'd like to plot the MAE of the regression prediction on the test data against the time it is predicting.

Ideally I'm hoping for a package in R for plotting the MAE graphically, essentially an MAE equivalent of the ROCR package for plotting the ROC curve?

$\endgroup$
  • $\begingroup$ Better to ask in general and then say an r-language routine would be appreciated. Otherwise, the question is not a stats question and should be closed. $\endgroup$ – Carl Apr 25 '17 at 20:55
  • $\begingroup$ Yes, OK, but why bother? When we model something, we want to know if it underestimates or overestimates. I leave terminal data out of the model, and then plot residuals; the withheld data minus the extrapolated model evaluated for its sample-times. When I want a single error measurement I look at the last point error, and the RMS of all extrapolated errors. $\endgroup$ – Carl Apr 26 '17 at 22:53
2
$\begingroup$

Of course you can plot lots of MAEs.

  • If you have a single time series you are predicting, you can simply plot a time series of your absolute error over the forecasting horizon. (Not much "mean" here.)

  • If you have multiple time series, e.g., predictions of rentals at different locations or of different products or car models, then you can average AEs over time series per future time bucket and then plot time series of MAEs.

    If so, I usually find it very enlightening to also plot the time series of quantiles of errors - sometimes the MAEs look fine, but the high quantiles (the "bad boys") are the ones you learn a lot from.

Here is an example:

horizon <- 10
nn <- 20

set.seed(1)
aes <- ts(matrix(exp(rnorm(horizon*nn,log(matrix(1:horizon,nrow=horizon,ncol=nn)+1),1)),
    nrow=horizon,ncol=nn))

foo <- ts(cbind(rowMeans(aes),t(apply(aes,1,quantile,c(0.1,0.9)))))
plot(foo[,1],ylim=range(foo),lwd=2,ylab="",main="Absolute Errors\n10%, Mean, 90%")
lines(foo[,2]); lines(foo[,3])

AEs

I am not aware of any R package to do this automatically, but as you see, it's not hard to munge and plot your errors by hand.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.