I would like to build a classifier which distinguishes between buyers and non-buyers based on user behavior. This data is highly imbalanced (0.009% for positive class), and I'm currently trying the
library(rpart) to build such a classifier (I'm also considering Random Forests, please make a comment if I should use it)
You can use the following code to load the dataframe:
df.train <- structure(list(result = c(0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 0L), totals_pageviews = c(3L, 1L, 1L, 2L, 7L, 12L, 6L, 1L, 1L, 15L), totals_sessionQualityDim = c(3L, 1L, 1L, 2L, 6L, 45L, 16L, 1L, 1L, 40L), device_deviceCategory = structure(c(2L, 1L, 2L, 3L, 1L, 1L, 1L, 1L, 1L, 1L), .Label = c("desktop", "mobile", "tablet"), class = "factor")), .Names = c("result", "pageviews", "quality", "device"), row.names = c(1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 9L, 10L), class = "data.frame")
This is just a sample dataset, I have more dimensions and rows. Other dimensions are categorical with 50+ levels.
Instantly, I notice that the result column is just an integer. This should infact be a factor, and I am using
method="class" in the
rpart() for this reason.
Here is my tree where I created a loss matrix, so that False Negatives are given a higher cost:
lossmatrix <- matrix(c(0,10,1,0), byrow=TRUE,nrow=2) mytree <- rpart(result ~.,data = df.train, method = "class", maxdepth=4, minsplit = 10, minbucket=5, parms = list(loss=lossmatrix))
Should I use
as.factor() on my factor variables? Or does
rpart() automatically fix them? It seems that R already knows which are factors by the picture above.
Next, I plotted the prediction tree:
pred.tree <- predict(mytree, newdata = df.test)
Checked the accuracy using:
This gave me precision: 0.009, recall: 1.000, F: 0.009, which looks like I'm predicting 1 to all situations...
I'm also not sure which class does
rpart()take as the positive class, because it could be that the algorithm is predicting 0 all the time (due to imbalance) which gives me good recall (since it correctly finds 'positive' classes) and bad precision, because I always predict 0.
Additionaly, in the guide I've followed
accuracy.meas(df.test$result, pred.tree[,1]) is substituted by
accuracy.meas(df.test$result, pred.tree[,2]) and if I exclude the
rpart() none of the above work, and I need to simply have
accuracy.meas(df.test$result, pred.tree), so I'm not sure about the indexes.
Lastly, I tried undersampling, oversampling, using ROSE() without lossmatrix, all without any improvement. ROC area is consistently over 0.95 for all variations, F score doesn't pass 0.1
#under sampling data_balanced_under <- ovun.sample(result ~ ., data = df.train, method = "under",N = 50190)$data table(data_balanced_under$result) # 0 1 #25095 25095 tree.under <- rpart(result ~ ., data = data_balanced_under, method="class") pred.tree.under <- predict(tree.under, newdata = df.test) accuracy.meas(df.test$result, pred.tree.under[,2]) #F 0.08, P=0.088, R=0.957 roc.curve(df.test$result, pred.tree.under[,2]) Area under the curve (AUC): 0.935