I'm trying to write a discretized version of Continuous Rank Probability Score (CRPS).

$$CRPS=\frac{1}{n_{ \mbox{cases}}}\sum_{i=1}^{\displaystyle n_{\mbox{cases}}} \left(F^f_i(x) - F^0_i(x)\right)^2dx$$

where $F^f_i(x)$ is the predicted CDF of case $i$ and $F^0_i(x)$ is the true CDF of case $i$.

I'm confused because Kaggle descretized this in their How Much Did it Rain competition as follows:

$$C = \frac{1}{70N} \sum_{N} \sum_{n=0}^{69} \left(\mathbb{P}(y \le n) -H(n -z)\right)^2$$

where $H(x)$ is the Heavyside step function: $H(x)=\begin{cases} 1, & \text{if }x \ge 0 \\ 0, & \text{otherwise} \end{cases}$.

I don't see how these two are equivalent. Can someone shed some light on this? Additionally, I have the following dataset (in R):

df <- data.frame(
  x = c(-2, -1.5, -1, -0.5, 0, 0.5, 1, 1.5, 2),
  PredCDF = c(0.061, 0.102, 0.184, 0.204, 0.449, 0.673, 0.776, 0.918, 1),
  ActualCDF = c(0.021, 0.043, 0.149, 0.298, 0.426, 0.638, 0.809, 0.915, 1)
     x PredCDF ActualCDF
1 -2.0   0.061     0.021
2 -1.5   0.102     0.043
3 -1.0   0.184     0.149
4 -0.5   0.204     0.298
5  0.0   0.449     0.426
6  0.5   0.673     0.638
7  1.0   0.776     0.809
8  1.5   0.918     0.915
9  2.0   1.000     1.000

If someone would take a stab at calculating the CRPScore for this data, that'd be very helpful.


Firstly, you have a 'typo' in the form of a missing integral in your mathematical formula.

$$CRPS=\frac{1}{n} \sum_{i=1}^{n} \int_{-\infty}^{\infty} \left(F^f_i(x) - F^0_i(x)\right)^2dx$$

Secondly, since the outer sum in both cases is essentially an average over the CRPS over all cases, let's just examine whether the base CRPS formula is equivalent for individual cases.

The answer is yes, but you seem to have misunderstood what is meant by the phrase "true CDF".

The CRPS is a method of scoring a probability function over a particular prediction, against the actual event. So, in weather terms, a prediction could be something like having a probability distribution over a range of temperatures, and the actual event was a single temperature.

The probability mass function of the actual event is a single spike of probability 1 at the actual value, and zero elsewhere, since this is a deterministic event. If it is a continuous variable, then equivalently it's density function is a delta function. We know the value with absolute certainty, therefore the probability that the actual event was not that value is zero. Equally, the actual CDF is essentially a heavyside function, i.e. a step from 0 to 1 that occurs at that particular temperature.

Therefore the formula above should now make since, on the understanding that $F_i^0(x) $ only takes the values 0 and 1 at any time. The "square" term essentially penalises at each point the extent to which your pdf has over- or under- estimated the prediction respectively. The Kaggle formulation simply makes this explicit, and replaces the integral with a summation over discrete points (i.e., as if we're dealing with a probability mass function instead of a probability density function)

As for the calculation in your example, it cannot be done, since, as I've pointed out above, you seem to have misunderstood what "Actual CDF" means. Since the values you give are not {0,1}, this is not the Actual CDF for that event.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.