2
$\begingroup$

I have a classification model which classifies user's bank transactions into two categories. From this model I produce precision and recall metrics.

I would like to understand the confidence around these precision and recall metrics. However, because the transactions in the model can belong to the same user, they are not necessarily independant. Because of this, what is the best way to calculate the confidence interval of the precision and recall metrics?

$\endgroup$

1 Answer 1

0
$\begingroup$

You need to account for the "repeated measurement" of a single user.

  • You can use multiple group shuffle test/train splits where you account for the groups of data generated by a single user
  • or you use block bootstrap to generate confidence intervals, where you block the predictions related to a single user together and draw all of those predictions together in the bootstrap step
$\endgroup$
3
  • $\begingroup$ This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. - From Review $\endgroup$
    – utobi
    Commented Feb 19 at 17:32
  • 1
    $\begingroup$ I added more details $\endgroup$
    – Ggjj11
    Commented Feb 19 at 18:01
  • 1
    $\begingroup$ Sounds much better now (+1). I'm not the downvoter, but it could be helpful to the OP or future readers facing a similar problem if you provide a reference or suitable pointers for further reading. $\endgroup$
    – utobi
    Commented Feb 19 at 21:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.