2
$\begingroup$

Introduction:


I would like to build a classifier which distinguishes between buyers and non-buyers based on user behavior. This data is highly imbalanced (0.009% for positive class), and I'm currently trying the rpart() from library(rpart) to build such a classifier (I'm also considering Random Forests, please make a comment if I should use it)

Dataset


You can use the following code to load the dataframe:

df.train <- structure(list(result = c(0L, 0L, 0L, 0L, 0L, 1L, 0L, 0L, 0L, 
0L), totals_pageviews = c(3L, 1L, 1L, 2L, 7L, 12L, 6L, 1L, 1L, 
15L), totals_sessionQualityDim = c(3L, 1L, 1L, 2L, 6L, 45L, 16L, 
1L, 1L, 40L), device_deviceCategory = structure(c(2L, 1L, 2L, 
3L, 1L, 1L, 1L, 1L, 1L, 1L), .Label = c("desktop", "mobile", 
"tablet"), class = "factor")), .Names = c("result", "pageviews", 
"quality", "device"), row.names = c(1L, 
2L, 3L, 4L, 5L, 6L, 7L, 8L, 
9L, 10L), class = "data.frame")

This is just a sample dataset, I have more dimensions and rows. Other dimensions are categorical with 50+ levels.

Here's the overview of the dataset... Data Overview

Instantly, I notice that the result column is just an integer. This should infact be a factor, and I am using method="class" in the rpart() for this reason.

Method


Here is my tree where I created a loss matrix, so that False Negatives are given a higher cost:

lossmatrix <- matrix(c(0,10,1,0), byrow=TRUE,nrow=2)
mytree <- rpart(result ~.,data = df.train, method = "class",
                maxdepth=4, minsplit = 10, minbucket=5,
                parms = list(loss=lossmatrix))

Should I use as.factor() on my factor variables? Or does rpart() automatically fix them? It seems that R already knows which are factors by the picture above.

Next, I plotted the prediction tree:

pred.tree <- predict(mytree, newdata = df.test)

Checked the accuracy using:

accuracy.meas(df.test$result, pred.tree[,1])

This gave me precision: 0.009, recall: 1.000, F: 0.009, which looks like I'm predicting 1 to all situations...

I'm also not sure which class does rpart() take as the positive class, because it could be that the algorithm is predicting 0 all the time (due to imbalance) which gives me good recall (since it correctly finds 'positive' classes) and bad precision, because I always predict 0.

Additionaly, in the guide I've followed accuracy.meas(df.test$result, pred.tree[,1]) is substituted by accuracy.meas(df.test$result, pred.tree[,2]) and if I exclude the method="class" from rpart() none of the above work, and I need to simply have accuracy.meas(df.test$result, pred.tree), so I'm not sure about the indexes.

Lastly, I tried undersampling, oversampling, using ROSE() without lossmatrix, all without any improvement. ROC area is consistently over 0.95 for all variations, F score doesn't pass 0.1

EDIT UNDERSTAMPLING:

#under sampling
data_balanced_under <- ovun.sample(result ~ ., data = df.train, method = "under",N = 50190)$data
table(data_balanced_under$result)
#    0     1 
#25095 25095 

tree.under <- rpart(result ~ ., data = data_balanced_under, method="class")
pred.tree.under <- predict(tree.under, newdata = df.test)

accuracy.meas(df.test$result, pred.tree.under[,2]) #F 0.08, P=0.088, R=0.957
roc.curve(df.test$result, pred.tree.under[,2])
Area under the curve (AUC): 0.935

ROC

$\endgroup$
  • $\begingroup$ I tried under sampling, 25095 positive class occurrences, 2893855 negative $\endgroup$ – GRS Nov 9 '17 at 16:15
  • $\begingroup$ If you undersampled (25095 positive, 25095 negative examples) there is no possible way to have an AUC > .95 and an F1 < .1. The math just doesn't work. $\endgroup$ – Zach Nov 9 '17 at 16:17
  • $\begingroup$ @Zach Please see my edit... I've double checked, everything seems to be fine $\endgroup$ – GRS Nov 9 '17 at 16:23
  • $\begingroup$ Post the confusion matrix of test results. $\endgroup$ – Zach Nov 9 '17 at 16:25
  • $\begingroup$ To get the confusion matrix, I've added type="class" to my prediction call. This gave me: table(pred.tree.under) 0 1 \\ 881230 9175 \\ This is the test set, so it's imbalanced $\endgroup$ – GRS Nov 9 '17 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.