I am trying to create a classification model that predicts whether a customer will enquire for a financial product based on some 250 independent variables. 98% of the variables are count variables and the remaining are continuous variables. The dependent variable has two values "yes" or "no" where 23% cases belonging to the "yes" class and 77% to the "no" class. There is class imbalance. The dataset has 217000 rows. So I think there are plenty of cases to train the model on. The count variables are right skewed, so I have applied the log(feature+1) transformation on them.

I started with 250 variables and then selected the top 20 variables as given by the varImp function of the caret package in R. When I create a model the NIR value of the model is closed to 77%, for certain threshold I get a good accuracy (as the model predicts the "no" cases correctly) which indicates the proportion of the negative class. My objective is to predict the positive class with high precision (TP/TP+FP).

In nutshell I have tried undersampling the majority class by taking 50-50 ratio while training and also penalizing the wrong decision made for the "positive" class by specifying the class.weight=c(1000000,1) in the train function of the caret package. But still the results are as good as a logistic regression model only.

Is there something more I can try while tuning or some parameters have been provided incorrectly or can I conclude that I need to work on better features.

The call to train the model looks like this:


tgrid = expand.grid(.mtry=15,.splitrule="gini",.min.node.size=c(10,15,50,100,200,500))


Random Forest does feature selection internally and is robust to redundant and irrelevant features. So while developing the model I would keep all feature. This might give a small increase in performance.

Try a boosting algorithm, they often perform slightly better and probably closest one can get to the 'limit' of your features. For example Gradient Boosted Trees / xgboost.

  • $\begingroup$ I tried boosting also but it gave a very little bump in terms of accuracy. However, I want a model with high precision as well as high sensitivity. $\endgroup$ – tushaR Jul 1 '18 at 3:59
  • $\begingroup$ Then I think you should attempt to construct some better features. Transform/combine existing data, include secondary data sources. Collect new data with better information. Also consider to which degree your problem actually is predictable, some randomness is expected in most real life situations $\endgroup$ – Jon Nordby Jul 2 '18 at 13:37
  • $\begingroup$ Interesting statement made by you: "consider to which degree your problem actually is predictable". How can I calculate or quantify this? $\endgroup$ – tushaR Jul 4 '18 at 5:59
  • $\begingroup$ I am not aware of any universal methods. Most correlation metrics assume linear relationship and thats not necessary for predictability.Maybe some entropy measures can be useful. You can use various Exploratory Data Analysis techniques to better understand the data you have. You can also attempt some 'dream design'. Imagine what kind of observations you would need in order to make perfect predictions. Likely there are some obstacles there, how big they are can reveal something about the nature of the problem $\endgroup$ – Jon Nordby Jul 4 '18 at 21:46
  • $\begingroup$ I am getting much better results using the h2o library's random forest. $\endgroup$ – tushaR Jul 9 '18 at 7:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.