# Appropriate error measure

Statistics is not realy my field of knowlegde but I am trying to find an answer for the following question:

Many cities nowadays have a bike share system. Suppose you were asked to predict how many bikes are rented on a given day. It is more expensive to disappoint a customer than it is to have bikes left over at the end of the day. What would be an appropriate error measure and why?

Help would be realy appreciated!

• I'm assuming that you're trying to predict the number of rentals as a continuous variable. In this case any error function that weights a positive difference between your ground truth and the prediction high than the other way around would generally fit that idea. If I find the time later I'll elaborate on this in an answer – deemel Jan 29 '18 at 14:03

You need an asymmetric error measure. This will lead to interesting consequences, by the way, such as that your optimal forecast will not be the mean (average) anymore.

I'll give you an example of an asymmetric measure: $$|\ln \frac{f(x)} y |$$

Your model forecasts $f(x)$, while the true demand is $y$. If you overestimate the number of bikes, then your loss is the unused capacity. If you underestimate then the loss is unobtained revenue plus dissatisfied customers, who may leave your service.

Suppose, you overestimated the demand by 90%, in this case the loss is $|\ln 1.9|\approx 0.64$, while if you underestimate by 90% your loss is $|\ln 0.1|\approx 2.3$, which is much larger.

The intuition behind this loss is as follows. Look at the Taylor approximation for small errors $|e=f(x)-y|<<1$: $$\ln \frac{f(x)} y=\ln \frac{y+e} y=\ln(1+\frac e y) \approx\frac e y$$ For small errors $e$ this loss function is similar to relative loss. However, for larger errors it becomes very asymmetric penalizing underforecasting more than overforecasting.

Note, that this particular loss will lead to the optimal forecast being a median instead of the usual mean.

Maybe you could model your demand for bikes as a probability distribution and then look at a low quantile, e.g. in 95% of the days, what would be the maximum demand? This concept would be related to the Value-at-risk idea.

EDIT: Now when it comes to assess the quality of the aforementioned probability distribution, you could use for example the Mean Absolute Error (MAE). For instance, you could estimate the distribution with some training data (maybe even using some predictors like subscribers, rain or sunshine), use that distribution to predict the bike demand for some test data and measure the mean of abs(predicted-observed test data). The Mean Absolute Error has an immediate interpretation: how far off is the prediction from the reality?

If this error is small enough, then you could use the quantile approach above to determine the bikes you need to provide.

In other words, I suggest to divide the problem in two: 1) find a good model for the distribution, as can be examined by the MAE, 2) Use a quantile approach to find the "safe" amount of bikes.

• Not a bad suggestion--but isn't the question asking for a different kind of answer? "What would be an appropriate error measure." – whuber Jan 29 '18 at 15:46
• Right. Expanded my answer. – Karsten W. Jan 29 '18 at 19:09
• Unfortunately, the MAE doesn't accomplish what is requested because it treats errors below and errors above symmetrically. Very closely related threads that shed light on this issue are stats.stackexchange.com/questions/251600/… and stats.stackexchange.com/questions/132622/… (more technical and general). – whuber Jan 29 '18 at 19:13
• OP needs an asymmetric loss – Aksakal Jan 29 '18 at 19:22