1
$\begingroup$

I have a train, validation and testset with distribution 70-20-10 percent resp. The label has a distribution of label A 85% and label B 15%. If the model predicts label B and this is wrong (FN)I give it a high penalty in the formula to determine the optimal threshold.

Because the amount of available data seems to be limited in my opinion I added more data to the data (consulting the business). This however shifted the distribution of label A-label B to 50%-50%.

The results were not too bad (important metrics NPV of 86% and specificity of 22%). This is better than random guessing and better than the current situation. However, somehow I am not satisfied with the results and want to improve it. I wonder whether the change of distribution changed my results.

So I left the enrichment of the data out. The distribution now is back to 85-15. The NPV is 66% and specificity is 4%. Also, the number of predictions for label B is almost 0 out of 1000(test set).

So, I wonder, does it mean that my features do not have or have limited influence? and thus the model (nn) relies on the distribtion?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.