How to improve neural network sensitivity with a lopsided binary outcome?

Question

I'm working on predicting (not explaining) a 0/1 outcome that generally has only about 10% "1"s (I'm not at liberty to name the variables). N ~40,000. Logistic regression proved unsatisfactory, both when using about 5-10 main effects and after I built in several interaction terms suggested by CHAID procedures. Sensitivity was ultimately only about 25%.

I then turned to neural networks (radial basis function networks, in SPSS). I was pretty shocked to see the program fail to classify any cases as "1"s. That is, sensitivity was zero. First question: Is this a common or understandable NN result under these conditions?

Next I tried randomly excluding a large number of "0" cases in the training set, bringing the fraction of "1"s up to about 40%. Now the program was able to correctly identify a halfway-decent number of cases in the training set, with sensitivity around 30%, but that dropped to 20% when the solution was applied to the test set, which once again consisted of only about 10% "1"s.

2nd question: How would you get around this problem?

Answer 1

Yes, this is common with an imbalance in training data and some types of relationships.

Suppose bad students pass a tough course with probability $0$, while good students pass the course with probability $1/3$. If the only information you get to observe is whether the student is good or bad, then your most accurate prediction is that the student will fail every time. You may learn from the training data that a good student is more likely to pass than a bad student, but you will never believe that a particular student is more likely to pass than to fail.

Is this really a problem? That depends on how you want to use the model. If you have to bet a dollar for each student on whether the student will pass or fail, it may be right to bet that each student will fail. If you feel it is more costly to predict A for something which is actually of class B than to predict B for something which is actually A, then you may want to incorporate that into the cost function during training. If you are trying to generate realistic-looking data, then you may want to use the model's outputs stochastically instead of generating the most likely outcome.

In some cases there is enough observable information, but the model is not learning this. For example, if you observe latitude and longitude, and try to classify the location as "Delaware" vs. "Not Delaware," then your classifier might first learn that Delaware is negligibly small. You can try things such as changing the cost function (such as from squared error to cross-entropy loss) which severely punishes assigning a low probability to the correct class. You can select a more balanced subset of the data. If you rebalance the data, you could include equal numbers of points in and out of Delaware, or you could focus on points which a simpler classifier believes are close to Delaware. This may trade accuracy in areas you don't believe are close to Delaware for accuracy near the known borders. If you concentrate on getting the border with Maryland right, you might miss the fact that Delaware isn't connected since it includes a bit of land across the Delaware River.