I am forecasting items and measuring the point forecast and distribution accuracy of numerous different models against actuals. To measure distribution accuracy I am using the continuous-ranked probability score (CRPS.)
If I only desire to know how part of my forecast distribution performed (all parts of the distribution above the median forecast), is there a way to do this?