Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Statistics is not realy my field of knowlegde but I am trying to find an answer for the following question:
Many cities nowadays have a bike share system. Suppose you were asked to predict how many bikes are rented on a given day. It is more expensive to disappoint a customer than it is to have bikes left over at the end of the day. What would be an appropriate error measure and why?
Help would be realy appreciated!
You need an asymmetric error measure. This will lead to interesting consequences, by the way, such as that your optimal forecast will not be the mean (average) anymore.
I'll give you an example of an asymmetric measure: $$|\ln \frac{f(x)} y |$$
Your model forecasts $f(x)$, while the true demand is $y$. If you overestimate the number of bikes, then your loss is the unused capacity. If you underestimate then the loss is unobtained revenue plus dissatisfied customers, who may leave your service.
Suppose, you overestimated the demand by 90%, in this case the loss is $|\ln 1.9|\approx 0.64$, while if you underestimate by 90% your loss is $|\ln 0.1|\approx 2.3$, which is much larger.
The intuition behind this loss is as follows. Look at the Taylor approximation for small errors $|e=f(x)-y|<<1$: $$\ln \frac{f(x)} y=\ln \frac{y+e} y=\ln(1+\frac e y) \approx\frac e y$$ For small errors $e$ this loss function is similar to relative loss. However, for larger errors it becomes very asymmetric penalizing underforecasting more than overforecasting.
Note, that this particular loss will lead to the optimal forecast being a median instead of the usual mean.
Maybe you could model your demand for bikes as a probability distribution and then look at a low quantile, e.g. in 95% of the days, what would be the maximum demand? This concept would be related to the Value-at-risk idea.
EDIT:
Now when it comes to assess the quality of the aforementioned probability distribution, you could use for example the Mean Absolute Error (MAE). For instance, you could estimate the distribution with some training data (maybe even using some predictors like subscribers, rain or sunshine), use that distribution to predict the bike demand for some test data and measure the mean of abs(predicted-observed test data). The Mean Absolute Error has an immediate interpretation: how far off is the prediction from the reality?
If this error is small enough, then you could use the quantile approach above to determine the bikes you need to provide.
In other words, I suggest to divide the problem in two: 1) find a good model for the distribution, as can be examined by the MAE, 2) Use a quantile approach to find the "safe" amount of bikes.
