Error measure that take the time of prediction into account?

Question

I am doing a machine learning project where I'm attempting to predict the arrival times of buses equipped with GPS at bus stops, and I am very new to statistics/data analysis. I am looking for an error measure that would take into account the time at which the prediction is made.

For instance, let's say I have a data point (data point 1) recorded at 3:00pm, with an actual arrival time at a particular bus stop of 3:02pm on the same day, and another data point (data point 2) recorded at 3:00pm, with an actual arrival time at the same bus stop of 3:39pm on the same day. Say I produce a model that predicts the arrival time of data point 1 is 3:05pm on the same day, and 3:42pm for data point 2 on the same day. So the data about this model might look like this in JSON format:

[{"time-of-day":"3:00pm","actual-arrival":"3:02pm","predicted-arrival":"3:05pm"},
 {"time-of-day":"3:00pm","actual-arrival":"3:39pm","predicted-arrival":"3:42pm"}]

So the predictions for data points 1 and 2 have the same mean absolute error (in minutes) of 3. However, it can be argued that the prediction made by the model on data point 2 is better than the prediction made on data point 1, since the prediction was made further ahead in time.

Is there a commonly used statistical error measure that takes into account how far ahead in time the prediction is made?

Answer 1

This is essentially looking at different forecasting horizons, which is a very common exercise in forecasting. No, there is no commonly accepted error measure that takes horizons into account. What people typically do is calculate separate errors per horizon and then report those separately.

The problem with calculating one grand unifying error or accuracy KPI is that it implicitly weights errors by how far out they are. How much worse is an absolute error of 5 at horizon $h_1$ compared to the same absolute error at horizon $h_2$? The problem mainly lies with the implicitness of this weighting. Much better to simply report errors by horizon and have the user decide which ones are how relevant.

So you could calculate errors for predictions up to 10 minutes ahead, 10-20 minutes ahead, and so on. The added wrinkle in your case is what you do if your predicted and actual buckets do not match, e.g., you predict that the bus will arrive in eight minutes, but it actually does arrive in twelve. You might want to look at buckets organized both by predicted and actual waiting time and see whether there are any differences in what you see.