3
$\begingroup$

My dataset holds two classes, which in the real world are distributed about 95%\5%. I trained a model which performs well when the distribution is 70%\30% in the train set and the test set as well. Are my results reliable? Note: I have done this because it was easier to extract data creating this kind of distribution. My accuracy is about 96% and precision 98%.

$\endgroup$
1
$\begingroup$

The best way to determine the reliability of your model is by testing it on a separate test set, with a distribution close to that of the real world. While it may be more work to procure this dataset, the results will allow you to determine the true reliability of your model.

Without the recall it would be difficult to extrapolate into your situation from here, but supposing it sits around 90%, in a real-world dataset of 1000 samples you would expect to find 45 true positive samples (mislabeling 5 as negative) as well as ~19 false positives (based on the 98% precision, 950 * .02 = 19). That would mean that in a real world case, for the positive class you would achieve an approximate 70% accuracy rate (45 / (45 + 19) = ~.70). Whether or not that is reliable enough depends on your use case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.