I am trying to create a classification model that predicts whether a customer will enquire for a financial product based on some 250 independent variables. 98% of the variables are count variables and the remaining are continuous variables. The dependent variable has two values "yes" or "no" where 23% cases belonging to the "yes" class and 77% to the "no" class. There is class imbalance. The dataset has 217000 rows. So I think there are plenty of cases to train the model on. The count variables are right skewed, so I have applied the log(feature+1) transformation on them.
I started with 250 variables and then selected the top 20 variables as given by the
varImp function of the
caret package in R. When I create a model the NIR value of the model is closed to 77%, for certain threshold I get a good accuracy (as the model predicts the "no" cases correctly) which indicates the proportion of the negative class. My objective is to predict the positive class with high precision (TP/TP+FP).
In nutshell I have tried undersampling the majority class by taking 50-50 ratio while training and also penalizing the wrong decision made for the "positive" class by specifying the
class.weight=c(1000000,1) in the train function of the caret package. But still the results are as good as a logistic regression model only.
Is there something more I can try while tuning or some parameters have been provided incorrectly or can I conclude that I need to work on better features.
The call to train the model looks like this:
trControl=trainControl(method="cv", number=10,classProbs=TRUE,search="grid", allowParallel=TRUE,verboseIter=TRUE) tgrid = expand.grid(.mtry=15,.splitrule="gini",.min.node.size=c(10,15,50,100,200,500)) train(trainset,y=trainset$response,method="ranger",metric="ROC",trControl= trControl,tuneGrid=tgrid,num.threads=6,verbose=TRUE,preProc=c("zv","nzv"), importance="impurity",num.trees=1500,class.weights=c(1000000,1))