1
$\begingroup$

I have an algorithm which is able to make a prediction and given a confidence interval ( ex. 0.95 ) it generates the upper and lower bound for the prediction. My question is, how can i get an accuracy score ( in [0,1] ) for my prediction given the confidence interval and the output lower and upper bounds ?

My only idea was to use the bounds to compute the percentage of maximum error for the prediction, but in this way the confidence interval is not exploited in the calculation. (ex. prediction=2, upper=3, lower=1, the maximum error is (prediction-lower)/prediction or (prediction-upper)/prediction )

$\endgroup$
2
  • $\begingroup$ I don't understand. Can you elaborate what is an "accuracy score"? What will this tell you? $\endgroup$ Apr 16 '20 at 7:38
  • $\begingroup$ It will tell you how confident is your model for the single prediction. I just want a measure of how much the model is confident for the prediction. As an example, if the model with 0.95 confidence interval generates a upper/lower bounds very close to the prediction i want the accuracy score to be high. On the other hand if the bounds generate a large interval i want the score to be low. $\endgroup$
    – Davide
    Apr 16 '20 at 7:46
0
$\begingroup$

This is not really done. Anyway, a simple example off the top of my head would be:

  1. Normalize relative on all the data, if that is acceptable to you

So a standard min-max scaling, get the max distance between lower and upper bounds for all the predictions, and scale each interval/distance by this max distance, this will give you a new value between 0 and 1. You have to repeat this if you get new data though, because you might get a new interval with larger distance.

  1. Normalize relative to the interval

So similar to what you proposed, do $\frac{Predicted-Lower}{Upper-Lower}$ to get a value between 0 and 1, again you scale on the distance between max and min, but for each prediction individually.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.