0
$\begingroup$

I've generated a probability of fail for each item(row) in a pandas df using a classifier.

I would like to check that the probabilities are reasonable given the positive class rate.

Say I know that there are 134 fails per year, and I have 1,000,000 rows in my df.

How would I go about this?

Would it be reasonable to estimate the probability that any 134 of the million rows will fail using the probabilities outputted from the model, say by...

  1. calculate # of combos of 134 in 1 million
  2. take sample of 10k combos of size 134 with replacement
  3. calculate prob of each of the 10k combos failing (multiply the 134 individual row probabilities in each combo)
  4. take mean of the 10k probabilities to get mean prob of any combo failing
  5. multiply by number of combos (a) to get total prob

...effectively using N choose K but with unequally likely combinations and using sample?

Probability of fail for an item is independent of other items, and already adjusted to reflect imbalanced set.

Thank you!

$\endgroup$

1 Answer 1

0
$\begingroup$

Note that a classifier does not yield probabilities but instead involves a forced binary choice. You must be talking about a probability estimation method. Logistic regression will handle the entire range of probabilities, even for rare events. But with only 134 events you have a very small effective sample size for estimating the contributions of multiple predictors. Don't consider sampling with replacement to develop the probability model. You will need to have only a handful of candidate predictive features with 134 events or will need to make heavy use of data reduction that is blinded to the outcome variable (unsupervised learning) as a first step.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.