Scoring Rule for Continuous Probability Prediction I have a question about choosing the right scoring rule. I am building a system which predicts the spatial (2D) probability of an event. The label data contains continuous values between 0 and 1, indicating the probability of the event for each pixel. I constructed my NN and applied a sigmoid at the end to pull the values between 0 and 1. I started using the Brier-score (MSE) as loss function but after pondering a bit about it, it doesn't seem like a 'fair' scoring system in my case.
Take a scenario with two pixels, one pixel has a correct label of 0.5 and the other of 1. Imagine that we predict 0 for the first pixel and for the second 0.5, both will have the same Brier-score of (0.5-1)^2. However, I feel that the first case is actually worse than the second. The second prediction is far from correct, but it's halfway in between wrong and correct, it could have performed much worse (here we have equaled the average error of an untrained model). In the first case we have made the worst prediction we could possibly make (here we did much worse than what untrained model would on average predict). Intuitively, the first prediction, which was the worst prediction possible, should have a worse score than the prediction that was halfway in-between of the second pixel.
The second function I was considering was log-loss. However, since we're working with continuous values, taking P(x) seems like a bad operation to do as every P(x) will be very small.
I have searched the relevant topics for existing posts but haven't found a satisfying answer (or I didn't understand it).
What would be a 'good' scoring rule in this case?
 A: We have established in the comments that you are not really looking for scoring-rules to assess predictive densities, but for error measures to assess point predictions.
One tool that forecasters very often use and that would probably be helpful to you would be to use the Mean Squared Error (MSE), as you are doing - but using it in a relative way against a benchmark. So you really want to check whether your prediction improves on this benchmark.
For instance, if your outcomes are always constrained to be between 0 and 1, then one very simple benchmark prediction would be a flat 0.5 across all pixels. So you would calculate the MSE of your predictions, as you have been doing (calculating squared errors per pixel, averaging across pixels). In the second step, you would calculate the MSE of this very simple benchmark prediction. In the third step, you would divide your prediction MSE by the benchmark MSE. If this ratio comes out greater than one, then you did worse than the very simple benchmark.
Note how this approach indeed penalizes cases where your prediction is on the wrong side of 0.5, compared to the actual: if the actual is 0.8 and you predicted 0.3, then the flat prediction of 0.5 across all pixels would indeed have been better.
Another, possibly even better, benchmark would be to take the average across all pixels in your training data. Maybe your pixels are all between 0 and 1 - but the average is 0.4, not 0.5. In this case, a "naive" prediction would also rather be 0.4. (Forecasters like to use exactly this kind of "naive" forecast, the historical mean, as a sanity check. If your extremely sophisticated method cannot even beat this simple benchmark, you really don't have anything to be proud of. You would be surprised to see how often this happens.)
