How is it possible to turn out with a highly accurate prediction when all records were classified the same way?

Question

We ran a CHAID decision tree model using the set up and process described in my related question here. We used the propensity scores to come up with a prediction. We measured the prediction at the end of the year simply as actuals/prediction and got 95% accuracy. This was last year. This year we have a lower accuracy at 80% but not too bad.

I am coming up to speed with what a previous analyst did. But when I inspected his model I noticed everyone was classifed the same: "stayer" (part of a binary target). The min and max of propensity scores ranged from .035 to .41. I don't understand why the model was so accurate as measured at the end of the year. It was great to have those results but it just doesn't seem logical to me.

QUESTIONS#1 Was this just a fluke (two years in a row!?!)? Or is there something I am missing with respect to intpreting classification results (confusion matrix) versus actual model performance?

I can post a link to a SPSS Modeler .str file (and related text files, all anonimized) if anyone wants to see it.

QUESTIONS#2: Do folks who use decision trees for prediction ever just use the classifications assigned (i.e. count number of target class) as opposed to propensity scores? I suppose if they were accurate you might do so (unlike our case). But I think propensity scores should be better to use in either case, right?

EDIT:

Everyone is classifed as 0 ("Stayer")

This is what I mean by accuracy and how it is being calculated

This is a sample of what my data looks like.

Answer 1

This happens when your data set is very unbalanced--it looks like about 93% of your data comes from one class: "stayers." Assuming your test data is similar, the simplest possible classifier (i.e., guess the majority class) should be able to get about proportion correct; a better classifier--with access to actual information--should be able to at least that well. Seen in that light, your 95%/85% accuracy is unfortunately somewhat less exciting.

So, how can you deal with unbalanced data? Obviously, the percent accuracy is a little misleading in this situation; looking at confusion matrices or other per-class measures is definitely the right thing to do (good move!).

Some classification algorithms allow you to provide a loss function that describes the relative "badness" of misclassifying an example. If you made misclassifying a leaver as a stayer much worse than misclassifying a stayer as a leaver, you get better results. However, I'm not sure if this is an option with CHAID. If not, you might be able to fake it by dropping examples from your training data until the classes are (roughly) balanced. This is often done by dropping random examples, but you might be able to do better by selectively choosing "representative" cases.

Answer 2

Maybe I'm misunderstanding the question, but it's not unusual for a dumb classifier to do well in a highly unbalanced situation. In this case, more than 90% of the cases appear to fall into one category, so you'd get 90% accuracy with the simple rule of guessing that category. Your actual classifier doesn't appear to do better than this.