# Understanding Brier Loss Composition

I built several models and measured the brier loss, calibration loss, and reliability loss with the direct model and a calibrated one. Now I try to interpret the results, but I cannot understand them in combination with the calibration plots.

My understanding is that the smaller the calibration loss the better the calibration. But how can I interpret the refinement loss? According to Wikipedia, "The second term is known as refinement. It is an aggregation of resolution and uncertainty and is related to the area under the ROC Curve" But the ROC score does not change for any model. However, the refinement loss is vastly different for the "Gradient" and "XGB."

I also cannot connect the calibration plot to the actual calibration loss. For me, the uncalibrated plots look much more calibrated, but the score is lower for the calibrated models. I assume that maybe the missing part in the uncalibrated plot has something to do with it.

I am grateful for any hints also recommendations for literature.

Calibrated
Brier Loss       Calibration Loss  Refinement Loss
Classifier
Dummy Strat    0.162253          0.000034         0.162218
XGB            0.150804          0.085912         0.064892

Uncalibrated
Brier Loss       Calibration Loss  Refinement Loss
Classifier
Dummy Strat    0.326294          0.164076         0.162218
XGB            0.151905          0.128606         0.023299


The first picture shows the calibration plot (Left: Uncalibrated / Right: Calibrated)

The second picture shows the distribution plot (Left: Uncalibrated / Right: Calibrated)