I have a train, validation and testset with distribution 70-20-10 percent resp. The label has a distribution of label A 85% and label B 15%. If the model predicts label B and this is wrong (FN)I give it a high penalty in the formula to determine the optimal threshold.

Because the amount of available data seems to be limited in my opinion I added more data to the data (consulting the business). This however shifted the distribution of label A-label B to 50%-50%.

The results were not too bad (important metrics NPV of 86% and specificity of 22%). This is better than random guessing and better than the current situation. However, somehow I am not satisfied with the results and want to improve it. I wonder whether the change of distribution changed my results.

So I left the enrichment of the data out. The distribution now is back to 85-15. The NPV is 66% and specificity is 4%. Also, the number of predictions for label B is almost 0 out of 1000(test set).

So, I wonder, does it mean that my features do not have or have limited influence? and thus the model (nn) relies on the distribtion?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.