Proving significantly better performance in binary forecasting

Question

I have two different algorithms that make forecasts for binary events. The observed result can either be 1 or 0 (like "rain" or "no rain"). The algorithms usually give a forecast in the 0.4-0.6 range.

For measuring error I take the difference between forecast and actual result to the power of 2. So if algorithm 1 said there was a 45% chance of raining and it did rain the resulting error from that observation would be (1-0.45)^2.

I do this for all observations and sum up the errors over all observations (giving me something like the Brier Score, with the only difference being that I'm not dividing by #of observations).

Suppose algorithm 1 has a lower sum of error than algorithm 2. How can I test whether algorithm 1 is significantly better than algorithm 2?

Answer 1

You could do a permutation test: re-sample the predictions to take the same number of predictions you originally had in your results for each algorithm, but randomly assigning the scores to algorithms while keeping the same number of positive and negative labelled sets. Calculate the difference in squared errors for each randomization. If you do this N times and k times you get a difference you observed or higher, the significance p-value for the difference in accuracy is $k/N$.

To simplify, suppose you were running the algorithms on the same data. For example, for data point $i$: the true label is $t_i$, algorithm 1 gives output $y^1_i$, and algorithm 2 gives $y^2_i$. The error cost for algorithm 1 is $e^1_i = (t_i - y^1_i)^2$, and $e^2_i = (t_i-y^2_i)^2$ for algorithm 2. Thus we have two vectors $\mathbf{e_1}=[e_1^1, e_2^1,...e_n^1]$, and $\mathbf{e_2}=[e_1^2, e_2^2,...e_n^2]$.

The difference in accuracy is $s=\|\mathbf{e_1}\|^2-\|\mathbf{e_2}\|^2$. Now let us just randomize byrandomly switching $e_i^1$ and $e_i^2$ for each $i$ with a probability of $0.5$. For each randomization $r$, you will get a difference in accuracy $s_r$. If you do this $N$ times, your p-value is $\frac{\sum_{r=0}^{N-1}\mathbf{1}_{|s_r|\ge s}}{N}$. This is the fraction of times you get an absolute value of the difference equal to or greater than that you observed in your experiment.