I am data modelling analyst in telecom company and now work on churn prediction model. I use decision tree algorithm with cross validation in SAS Enterprise Miner. The results are satisfactory as I get high percentage of predicted churners in top percentiles of predictions ( 55% true in top 3%). But recently I added some new variables to the model hoping that the accuracy will increase but it decreased instead. Can you please help me understand what happened and why new variables had bad impact on the model? Also what can I do to improve the model?
The new variables might allow the model to overfit on the training dataset and affect generalisation on unseen data.
Sometimes it can happen when the new variables are almost unique and identify the samples by themselves (timestamp, id, something noisy where the model learns the noise, ...). If the problem is noise in the variable, quantisation of the values might help.