Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm working on predicting (not explaining) a 0/1 outcome that generally has only about 10% "1"s (I'm not at liberty to name the variables). N ~40,000. Logistic regression proved unsatisfactory, both when using about 5-10 main effects and after I built in several interaction terms suggested by CHAID procedures. Sensitivity was ultimately only about 25%.

I then turned to neural networks (radial basis function networks, in SPSS). I was pretty shocked to see the program fail to classify any cases as "1"s. That is, sensitivity was zero. First question: Is this a common or understandable NN result under these conditions?

Next I tried randomly excluding a large number of "0" cases in the training set, bringing the fraction of "1"s up to about 40%. Now the program was able to correctly identify a halfway-decent number of cases in the training set, with sensitivity around 30%, but that dropped to 20% when the solution was applied to the test set, which once again consisted of only about 10% "1"s.

2nd question: How would you get around this problem?

share|improve this question
add comment

1 Answer

Yes, this is common with an imbalance in training data and some types of relationships.

Suppose bad students pass a tough course with probability $0$, while good students pass the course with probability $1/3$. If the only information you get to observe is whether the student is good or bad, then your most accurate prediction is that the student will fail every time. You may learn from the training data that a good student is more likely to pass than a bad student, but you will never believe that a particular student is more likely to pass than to fail.

Is this really a problem? That depends on how you want to use the model. If you have to bet a dollar for each student on whether the student will pass or fail, it may be right to bet that each student will fail. If you feel it is more costly to predict A for something which is actually of class B than to predict B for something which is actually A, then you may want to incorporate that into the cost function during training. If you are trying to generate realistic-looking data, then you may want to use the model's outputs stochastically instead of generating the most likely outcome.

In some cases there is enough observable information, but the model is not learning this. For example, if you observe latitude and longitude, and try to classify the location as "Delaware" vs. "Not Delaware," then your classifier might first learn that Delaware is negligibly small. You can try things such as changing the cost function (such as from squared error to cross-entropy loss) which severely punishes assigning a low probability to the correct class. You can select a more balanced subset of the data. If you rebalance the data, you could include equal numbers of points in and out of Delaware, or you could focus on points which a simpler classifier believes are close to Delaware. This may trade accuracy in areas you don't believe are close to Delaware for accuracy near the known borders. If you concentrate on getting the border with Maryland right, you might miss the fact that Delaware isn't connected since it includes a bit of land across the Delaware River.

share|improve this answer
    
Thank you. It seems that cost function options come into play with multilayer perceptron but not with radial basis function networks. I tried a variety of such functions in MP but still sensitivity = 0, at least if I relied on the NN predicted value (classification) :-( Anyway, could you exlain what you mean by "use the model's outputs stochastically instead of generating the most likely outcome"? –  rolando2 Dec 24 '12 at 15:05
    
If you are trying to fill in missing target values (to generate data) then instead of filling in the most likely, you could fill in a random value whose probability distribution is given by your model. If your model says that target value A happens $35\%$ of the time, then you fill in A $35\%$ of the time. In some situations that is much better than filling in the most likely value $100\%$ of the time. E might be the most common letter in English but not many sentences consist of only Es. –  Douglas Zare Dec 24 '12 at 16:52
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.