0
$\begingroup$

I have a model which has an accuracy of $A$. It makes a prediction with $Y$ confidence about a number of samples. Samples are observed many times, but not all are observed equally. So my data looks like:

Model Accuracy (A)    Model Prediction Confidence (Y)        Sample ID
0.80                              0.90                            1
0.80                              0.95                            1
0.80                              0.70                            1
0.80                              0.88                            2
0.80                              0.92                            2
0.80                              0.80                            3
0.80                              0.93                            4
0.80                              0.85                            4
0.80                              0.60                            4
0.80                              0.97                            4

I'd like to restrict my analysis to samples that I'm 95% sure are real. How many times must I have observed a sample for it to fall within this range, if I filter the model prediction to be strictly above a determined $X$.

My thinking here is that if I lower $X$ I'll have more samples in my data so despite being less confident in them perhaps that is outweighed by the confidence I get of having a greater number?

It's not possible for me to bootstrap the model.

Thank you for your help.

Improve this question
$\endgroup$
2
  • $\begingroup$ I was thinking perhaps the thing to do is for a threshold $X$ to calculate the ratio of log odds to get the required number of observations $N$: $\lceil \frac{\ln(0.95 / (1 - 0.95))}{\ln(X / (1 - X))} \rceil$ And somehow incorporate the probability $A$ $\endgroup$
    – donkey
    Commented Nov 22 at 23:06
  • $\begingroup$ I could just do $X*A$ in my log odds. $\endgroup$
    – donkey
    Commented Nov 22 at 23:12

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.