1
$\begingroup$

I have two different algorithms that make forecasts for binary events. The observed result can either be 1 or 0 (like "rain" or "no rain"). The algorithms usually give a forecast in the 0.4-0.6 range.

For measuring error I take the difference between forecast and actual result to the power of 2. So if algorithm 1 said there was a 45% chance of raining and it did rain the resulting error from that observation would be (1-0.45)^2.

I do this for all observations and sum up the errors over all observations (giving me something like the Brier Score, with the only difference being that I'm not dividing by #of observations).

Suppose algorithm 1 has a lower sum of error than algorithm 2. How can I test whether algorithm 1 is significantly better than algorithm 2?

$\endgroup$
1
1
$\begingroup$

You could do a permutation test: re-sample the predictions to take the same number of predictions you originally had in your results for each algorithm, but randomly assigning the scores to algorithms while keeping the same number of positive and negative labelled sets. Calculate the difference in squared errors for each randomization. If you do this N times and k times you get a difference you observed or higher, the significance p-value for the difference in accuracy is $k/N$.

To simplify, suppose you were running the algorithms on the same data. For example, for data point $i$: the true label is $t_i$, algorithm 1 gives output $y^1_i$, and algorithm 2 gives $y^2_i$. The error cost for algorithm 1 is $e^1_i = (t_i - y^1_i)^2$, and $e^2_i = (t_i-y^2_i)^2$ for algorithm 2. Thus we have two vectors $\mathbf{e_1}=[e_1^1, e_2^1,...e_n^1]$, and $\mathbf{e_2}=[e_1^2, e_2^2,...e_n^2]$.

The difference in accuracy is $s=\|\mathbf{e_1}\|^2-\|\mathbf{e_2}\|^2$. Now let us just randomize byrandomly switching $e_i^1$ and $e_i^2$ for each $i$ with a probability of $0.5$. For each randomization $r$, you will get a difference in accuracy $s_r$. If you do this $N$ times, your p-value is $\frac{\sum_{r=0}^{N-1}\mathbf{1}_{|s_r|\ge s}}{N}$. This is the fraction of times you get an absolute value of the difference equal to or greater than that you observed in your experiment.

$\endgroup$
2
  • $\begingroup$ Thank you so far but I don't understand everything you wrote. Can't I just resample the observations, compute the error for both algorithms. Then if algorithm 1 is better in 95% of the redraws, it is significantly better? Do I really need to keep the same number of positive and negative labelled sets? What is "difference you observed or higher"? $\endgroup$ – Randy M. Aug 15 '11 at 15:53
  • $\begingroup$ @Randy, not exactly. The idea is to see if you can get a similar difference in accuracy by chance, if the two algorithms were doing a similar job. I have edited my answer to explain in more detail and give a step by step algorithm. You can ask if there is anything not clear and read up on p-values and exact tests on wikipedia. $\endgroup$ – highBandWidth Aug 15 '11 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.