I have a train, validation and testset with distribution 70-20-10 percent resp. The label has a distribution of label A 85% and label B 15%. If the model predicts label B and this is wrong (FN)I give it a high penalty in the formula to determine the optimal threshold.
Because the amount of available data seems to be limited in my opinion I added more data to the data (consulting the business). This however shifted the distribution of label A-label B to 50%-50%.
The results were not too bad (important metrics NPV of 86% and specificity of 22%). This is better than random guessing and better than the current situation. However, somehow I am not satisfied with the results and want to improve it. I wonder whether the change of distribution changed my results.
So I left the enrichment of the data out. The distribution now is back to 85-15. The NPV is 66% and specificity is 4%. Also, the number of predictions for label B is almost 0 out of 1000(test set).
So, I wonder, does it mean that my features do not have or have limited influence? and thus the model (nn) relies on the distribtion?