Given datasets consisting of a daily prediction and confidence percentage for each of a small number of binary outcomes, what is the proper way to calculate the forecast error of each series and of the data as a whole?
For example, say we are discussing today whether or not a given location will receive at least one quarter inch of snow on 3/1/2015, and there is a predictor that each day in February gives a new estimate of the chance from 0% to 100% that it will snow on 3/1/2015.
Now if there were a dozen locations we were doing this for, we could take the last prediction (from 2/29/2015) for each location as a decimal number from 0-1 (0% to 100%) for the forecast value, use either 0 or 1 for whether it snowed at each location as the actual value, and apply one of the forecast error formulas such as MAE, MSE, etc. (MPE/MAPE wouldn't work since we would have actual values of 0).
However, this doesn't give us any information about the error of the daily prediction from the rest of the month. Would it make sense to generate a forecast error for each series by comparing each daily prediction to 0/1 and applying MAE to the whole series and then comparing that to a daily 50% forecast to obtain a predictive value?
It seems this makes sense for some straightforward cases:
A prediction that was 100% sure of the right outcome the first day and stayed at 100% would have an MAE of 0 and we would have high confidence that future forecasts from this predictor were correct.
A prediction that was 100% sure of the wrong outcome the first day and stayed at 100% would have an MAE of 1 and we would have high confidence that future forecasts from this predictor were the exact opposite of the actual value.
A prediction that refused to guess and gave a 50% chance every day would have an MAE of 0.5, giving us no confidence in the prediction.
A prediction that flipped a coin every day and selected 0% or 100% based on the flip would also have an expected MAE of 0.5 (50% of days would be an error of 0, 50% of days would be an error of 1, making the MAE 0.5), again giving us no confidence in the prediction.
A prediction that started at 50% on day 1 and linearly increased to 100% on the last day would have an MAE of 0.25, making it better than pure chance.
In addition, what would be the best way to combine the forecast errors from each series to obtain an overall forecast error for the predictor over the whole dataset? Some series may have more data points and from farther out than others: one series might be giving predictions for a date in March 2016 on each day in February 2016, while another series has predictions for a date in January 2016 on each day starting in October 2015.
Or am I going about this all wrong and need to apply some type of time weighting based on how far out the forecast was? On the one hand, we would expect there to be more error the farther in the past we go; on the other hand it seems we should "reward" forecasts that correctly realized that and gave closer to 50% on days way in the past as opposed to forecasts that overstated their confidence and were wrong.