If I understand your question correctly, you might want to check out this question. As I explained there, one way of assessing the calibration of probability predictions is with a scoring rule. A common example of a scoring rule is the Brier score: $$BS = \frac{1}{N}\sum\limits _{t=1}^{N}(f_t-o_t)^2$$ where $f_t$ is the forecasted probability of the event happening and $o_t$ is 1 if the event did happen and 0 if it did not.

Of course the type of scoring rule you choose might depend on what type of event you are trying to predict. However, this should give you some ideas to research further.

Perfect prediction with the Brier score would actually be 0 though, so you could take $1 - BS$ if that quality is important to you. Note though that the other extreme score (0 or 1 depending upon whether you decide to flip the Brier score) actually would not be pure randomness but rather would represent getting the wrong answer every time.

If I understand your question correctly, you might want to check out this question. As I explained there, one way of assessing the calibration of probability predictions is with a scoring rule. A common example of a scoring rule is the Brier score: $$BS = \frac{1}{N}\sum\limits _{t=1}^{N}(f_t-o_t)^2$$ where $f_t$ is the forecasted probability of the event happening and $o_t$ is 1 if the event did happen and 0 if it did not.

Of course the type of scoring rule you choose might depend on what type of event you are trying to predict. However, this should give you some ideas to research further.

Perfect prediction with the Brier score would actually be 0 though, so you could take $1 - BS$ if that quality is important to you. Note though that the other extreme score (0 or 1 depending upon whether you decide to flip the Brier score) actually would not be pure randomness but rather would represent getting the wrong answer every time.