We inherited a turnover prediction process from a previous analyst. The tool we are using is SPSS Modeler. The model being used is CHAID decision tree. The target variable is binary: "stayer" vs "departer" (e.g. person stays with company, person leaves the company). The process is detailed below under BACKGROUND.
I am looking for this specific feedback from the CV community:
- What do you see inheritly wrong with the below process?
- What are some better ways of arriving at a predication using a decision tree classification model? ( I am only looking at decision tree models because I am new to data mining and predictive modeling and these models are easier to understand)
I should mention that accuracy has so not been a problem thus far. But what prompted this question is the fact that everyone is classified as "departer" (100% false positives). I posted a separate question about that here.
I can post a link to a SPSS Modeler .str file (and related text files, all anonimized) if anyone wants to see it.
BACKGROUND
Task: Predict number of terminations for three of our directorates (e.g ~ “business units”) for upcoming fiscal year using two years historical terminations and headcount data.
Datafile: Employee data with several demographic fields. Every record is an employee that either stayed during the entire two year period or terminated (i.e. "stayer" vs "departer")
Our current process involves runing a CHAID Model interactively using SPSS Modeler for a recent known time period. For example 2009-2010 (file setup as decribed above).
Sidenote: This is the "training" data. The data is not partioned into "training" and "testing" data in the traditional classification sense. This is mainly because the count of "departers" is relatively small compared (1:14) to "stayers".
We do this as follows:
1) CHAID interactive model is run iteratively (trial and error), each time measuring the performance by this subprocess:
a. Apply the model to beginning of year population (e.g. 2011 in this case) to come up with propensity scores for each individual.
b. Sum the propensity scores by organization and divide by two. We divide by two since terminations being used to train the model cover a two year period but we are only interested in projecting a year out. This produces predictions by directorate.
c. Once the accuracy is above 85% move on to next step.
2) Build new model using model from #1 as a guide . New model will be similar to one in #1--business knowledge and some guesswork is applied to adjust splits...but this part is minimal.
3) Apply final model to beginning of year population (e.g. 2012..assuming that's the upcoming year) to come up with final predications by directorate.
BTW I tried looking for similar posts on model deployment process but only found posts such as the following which didn't hit the mark for me help-with-deploying-a-model
