We ran a CHAID decision tree model using the set up and process described in my related question here. We used the propensity scores to come up with a prediction. We measured the prediction at the end of the year simply as actuals/prediction and got 95% accuracy. This was last year. This year we have a lower accuracy at 80% but not too bad.
I am coming up to speed with what a previous analyst did. But when I inspected his model I noticed everyone was classifed the same: "stayer" (part of a binary target). The min and max of propensity scores ranged from .035 to .41. I don't understand why the model was so accurate as measured at the end of the year. It was great to have those results but it just doesn't seem logical to me.
QUESTIONS#1 Was this just a fluke (two years in a row!?!)? Or is there something I am missing with respect to intpreting classification results (confusion matrix) versus actual model performance?
I can post a link to a SPSS Modeler .str file (and related text files, all anonimized) if anyone wants to see it.
QUESTIONS#2: Do folks who use decision trees for prediction ever just use the classifications assigned (i.e. count number of target class) as opposed to propensity scores? I suppose if they were accurate you might do so (unlike our case). But I think propensity scores should be better to use in either case, right?
Everyone is classifed as 0 ("Stayer")
This is what I mean by accuracy and how it is being calculated
This is a sample of what my data looks like.