I have two different algorithms that make forecasts for binary events. The observed result can either be 1 or 0 (like "rain" or "no rain"). The algorithms usually give a forecast in the 0.4-0.6 range.
For measuring error I take the difference between forecast and actual result to the power of 2. So if algorithm 1 said there was a 45% chance of raining and it did rain the resulting error from that observation would be (1-0.45)^2.
I do this for all observations and sum up the errors over all observations (giving me something like the Brier Score, with the only difference being that I'm not dividing by #of observations).
Suppose algorithm 1 has a lower sum of error than algorithm 2. How can I test whether algorithm 1 is significantly better than algorithm 2?