# Deployment process for Classification models (i.e. decision trees)

We inherited a turnover prediction process from a previous analyst. The tool we are using is SPSS Modeler. The model being used is CHAID decision tree. The target variable is binary: "stayer" vs "departer" (e.g. person stays with company, person leaves the company). The process is detailed below under BACKGROUND.

I am looking for this specific feedback from the CV community:

1. What do you see inheritly wrong with the below process?
2. What are some better ways of arriving at a predication using a decision tree classification model? ( I am only looking at decision tree models because I am new to data mining and predictive modeling and these models are easier to understand)

I should mention that accuracy has so not been a problem thus far. But what prompted this question is the fact that everyone is classified as "departer" (100% false positives). I posted a separate question about that here.

I can post a link to a SPSS Modeler .str file (and related text files, all anonimized) if anyone wants to see it.

BACKGROUND

Task: Predict number of terminations for three of our directorates (e.g ~ “business units”) for upcoming fiscal year using two years historical terminations and headcount data.

Datafile: Employee data with several demographic fields. Every record is an employee that either stayed during the entire two year period or terminated (i.e. "stayer" vs "departer")

Our current process involves runing a CHAID Model interactively using SPSS Modeler for a recent known time period. For example 2009-2010 (file setup as decribed above).

Sidenote: This is the "training" data. The data is not partioned into "training" and "testing" data in the traditional classification sense. This is mainly because the count of "departers" is relatively small compared (1:14) to "stayers".

We do this as follows:

1) CHAID interactive model is run iteratively (trial and error), each time measuring the performance by this subprocess:

a. Apply the model to beginning of year population (e.g. 2011 in this case) to come up with propensity scores for each individual.

b. Sum the propensity scores by organization and divide by two. We divide by two since terminations being used to train the model cover a two year period but we are only interested in projecting a year out. This produces predictions by directorate.

c. Once the accuracy is above 85% move on to next step.

2) Build new model using model from #1 as a guide . New model will be similar to one in #1--business knowledge and some guesswork is applied to adjust splits...but this part is minimal.

3) Apply final model to beginning of year population (e.g. 2012..assuming that's the upcoming year) to come up with final predications by directorate.

BTW I tried looking for similar posts on model deployment process but only found posts such as the following which didn't hit the mark for me help-with-deploying-a-model

-
First, if you try different models until you hit something that is better than $X$% (in your case $X=85$), you can not in general expect this accuracy on your test set, due to over fitting. Second, if you only need a 85% accuracy, and over 85% of your data is "stayer", then it will be sufficient to classify everyone as "stayer". Third, if your features (demographics, I guess?) are not informative enough, this may be the best that you can do. You can probably do better, but I'm not well versed enough in decision trees or your data to comment on that. –  MLS Aug 1 '12 at 15:43
What is most important to us is accurately identifying and classifying records as "departers" rather than ""stayers" but everyone is classified as "stayers" including the actual "departers" in the data. –  daniellopez46 Aug 1 '12 at 16:24