1
$\begingroup$

I am doing a classification modelling(using R and random forests) for a website where only 2% of the visitors convert. Now given the behavior and attributes of a visitor I want to predict probability of conversion.

The data collection process was pretty complicated and finally I had 3 month data available for 3166 cases of conversions & 10,849 non conversions. Normally, I know that the training & test data should have the same proportion of classes.

However, I wanted to use most of the "converted" data to train the model. I randomly took out 100 cases of conversions & 4000 cases of non-conversions to give me 1:40 ratio. This would be my test data.

For training, I took the remaining data which was approximately 1:2 ratio.

After training using Random Forests, when testing, I am getting decent results in terms of sensitivity & specificity but precision is low around 7-8%

What would be the possible repercussions of my approach? I wanted to get this sorted before I begin fine-tuning my model.

I did not do up-sampling/down-sampling or synthetic data generation because they would also ultimately try to balance my data when training the model but the test data would still reflect the true world scenario.

Any advice would be much appreciated.

EDIT 1: After the response by Fernando & DarXider, I tried the following 2 things:

a.) Took 200 Positive Class with all of Negative class. Then trained the models separately until Positive class is exhausted. Each model then predicted on the test data, their votes were counted and final probabilities calculated.

b.) Similar to the above except here 200 cases of Negative class were sampled with all of Positive Class.

However, the problem remains when testing. In case of (a) most trees vote for negative class when testing and vice-versa in case of (b).

I will try other suggestions and maybe other techniques and see what happens. I have already started the process of getting more data.. "Fingers crossed"

Code snippet for (a) is below.In case I have made any errors please do tell.. I know its a little inefficient code but am still learning :)

    tr_conv = trdata[trdata$Converted==1,]      ## converted & non-converted
    tr_nc = trdata[trdata$Converted == 0,]

    numr = nrow(tr_conv)         ##calculatin the number of rows
    min_size = 200               ##sample size
    temp = data.frame()          ##empty data frame to store the results
    temp = NA

    while(numr>0){

    a = ifelse((numr - min_size) < min_size,numr,min_size)# sample size
    rm(.Random.seed, envir=globalenv()) ## reset the seed each time
    k = sample(x = 1:numr,size = a)
    tr_conv1 = tr_conv[k,]            ## this sample will be trained on
    tr_conv = tr_conv[-k,]
    numr = nrow(tr_conv)

    # combining with non-converted training data

    comb1 = rbind.data.frame(tr_conv1,tr_nc)
    r1 = randomForest(x = comb1[,1:24],y = comb1$Converted,ntree = 2000,
					mtry = 6,strata = comb1$Converted,norm.votes = FALSE)
    pd = predict(object = r1,norm.votes = F,newdata = tsdata[,1:24]
         ,type = "vote")      ## tsdata is the 4100 test cases
    vt1 = data.frame(pd)
    vt1 = data.frame(vt1[,-1])
    temp = cbind(temp,vt1)

    }
    temp1 = temp[,-1]
    temp1$Yes = rowSums(temp1)    ## total number of yes votes
    tvc = 2000*(ncol(temp1)-1)    ## calculate the total votes cast
    temp1$No = abs(temp1$Yes - tvc)  ## total number of no votes
    myvotes = temp1[,c(16,17)]
    myvotes$yes_prob = myvotes$Yes/tvc  ## yes probabilit calculate
    myvotes$no_prob = 1 - myvotes$yes_prob  ## no probability calculate
    threshold = 0.5
    myvotes$prediction = ifelse(myvotes$yes_prob > threshold,1,0)
    j = confusionMatrix(data =myvotes$prediction,
        reference = tsdata$Converted,positive = "1",
        dnn =c("pred","actual"))
$\endgroup$
  • $\begingroup$ This is confusing. Your original dataset is 10849:3166, or ~3.4:1, is that right? $\endgroup$ – Fernando Mar 1 '17 at 18:18
  • $\begingroup$ @feranando.. yes that is correct but that is because of limitations in collecting the data. It is a huge process for the website as lots of visitors have to be filtered out and hit-level data from Big Query is then downloaded. That is why the original data set has that ratio but the actual conversions observed on the website is around 2%. Iam sorry for the confusion. $\endgroup$ – pankaj negi Mar 1 '17 at 19:41
  • $\begingroup$ If the real target rate is 2%, but your data say it's ~29%, there's a problem. You could downsample, but you may miss some patterns. Are you sure this is not a data processing/architeture problem? Either get the whole data or change the concept/thing you're modelling. If you squeeze the model to acomodate this, it will overfit. $\endgroup$ – Fernando Mar 1 '17 at 19:52
  • $\begingroup$ I will give a try to down sampling and darXider''s suggestions. However, in parallel I will also try to get the data once more.but to maintain the 2% in my training data while trying to get as many "conversion points will mean that the entire data I will be working with will blow up. $\endgroup$ – pankaj negi Mar 1 '17 at 20:05
  • $\begingroup$ @ Fernando From the ISLR book [video here]( youtube.com/…) they mention that when using Logistic regression we can later adjust the estimated intercept. Is this useful in my case and how could it be brought over to Random forests or I should try logistic regression. Thnx for your help $\endgroup$ – pankaj negi Mar 1 '17 at 20:31
1
$\begingroup$

If I understood correctly, the real-world scenario has a converted to non-converted ratio of 1:49, which is similar to that for your test data which is 1:40; so far, so good! However, the data you use for training your model has said ratio of about 1:2. Therein lies your problem, and it causes you to have lots of false positives and, thus, bad precision when you apply your trained model to the test data.

I suggest that you have a look at this paper, Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure, published in Neural Computation 14, 21–41 (2001) by Saerens et al. which proposes a method to adjust the posterior probabilities $p(y|x)$ -- obtained by applying your trained model to the test data, where $x$ represents your data and $y = 0, 1$ is the class label -- in a scenario where the proportions of classes, $p(y = 0)$ and $p(y = 1)$, differ between the training and test sets.

You could also try building an ensemble of classifiers in the spirit of the BalanceCascade method (see Exploratory Undersampling for Class-Imbalance Learning by Xu-Ying Liu, Jianxin Wu, and Zhi-Hua Zhou in IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS—PART B: CYBERNETICS, VOL. 39, NO. 2, APRIL 2009):

  1. For the first classifier, randomly sample a subset of the minority class from the training set that gives you the said ratio of approximately 1:40 when used with the entire majority class
  2. Train a classifier using this data (using, e.g., the argument class_weight = 'balanced' if you are using scikit-learn).
  3. Setting aside the selected minority samples from the pool of training minority class, perform the next subsampling to arrive at the second dataset.
  4. Used this set to train your second next classifier.
  5. Repeat until you have exhausted the entirety of the minority class members.

Finally, you will have a set of classifiers, trained on non-overlapping subsets of the minority class. You can then combine their predictions (e.g., using majority vote or weighted average) to arrive at the final prediction.

Having said this, the cascade-and-ensemble approach might not be effective in your case since each classifier will be trained on approximately 200 data points, which is a small number, which could lead to low generalization score, but YMMV; it might be worth trying.

$\endgroup$
  • $\begingroup$ thank you...I will try this and post an update here soon $\endgroup$ – pankaj negi Mar 1 '17 at 19:42
  • $\begingroup$ R randomForest package has a parameter called strata which handles that. $\endgroup$ – Fernando Mar 1 '17 at 20:42
  • $\begingroup$ I used the strata parameter but how to adjust it to use 1:40 instead of the default ie. the ratio to the 1 to 0 in the training data. $\endgroup$ – pankaj negi Mar 1 '17 at 20:49
  • $\begingroup$ Use the sampsize parameter, but I don't think you should pass 1:40, it should be randomForest(..., strata=train$y, sampsize=c(3166, 3166), nodesize=5). Assuming y is a 0/1 factor variable. $\endgroup$ – Fernando Mar 1 '17 at 20:56
1
$\begingroup$

Another option is to reuse the minority class: suppose your entire dataset has 10000 rows and you have 200 positive samples (exact 50:1 proportion).

What you can do (I'm not claiming this is a 'good' solution):

  1. Create two datasets: P with all "1" examples and N with all "0" examples.
  2. Shuffle the N dataset.
  3. Fit 50 different models like this:

model_1 uses all of P and samples 1 to 200 of N.

model_2 uses all of P and samples 201 to 400 of N.

...

model_50 uses all of P and samples 9801 to 10000 of N.

When applying this model just take the average of these 50 models. This may or may not work, depending on your data. To validate you could set aside some of the data and apply the same idea.

$\endgroup$
  • $\begingroup$ this is good. Let me get back to you with the results. $\endgroup$ – pankaj negi Mar 1 '17 at 20:57
  • $\begingroup$ This is the actually the EasyEnsemble method from the paper Exploratory Undersampling for Class-Imbalance Learning, which I have mentioned in my answer above and which contains the BalanceCascade method which I have referred to. It usually works well, depending on the dataset, of course. The problem which you might face while using this method is, however, the same problem you are currently facing, that is, the ratio of the P to N in the 50 training sets is 1:1, but each trained model will be independently applied to a test set which has a ratio of 1:40. again, YMMV. $\endgroup$ – darXider Mar 1 '17 at 21:03
  • $\begingroup$ BalanceCascade is a cool name! OP has 3166 P samples for each sub-model, so assuming there's a pattern, the P:N ratio on the test set doesn't matter, what matters is the hidden pattern on these samples (3166 may or may not be enough). $\endgroup$ – Fernando Mar 1 '17 at 21:13
  • $\begingroup$ The OP is already using a training set with a 1:2 ratio of P to N classes (~ 3000 P's and ~ 6000 of N's); it's close enough to the 1:1 ratio used in EasyEnsemble (your answer) that if there were such a pattern in the data, the OP would see better scores when applying the ensemble model to the test data. Maybe ~ 3000 P examples is still not enough, but my guess is that the 1:40 ratio of P to N in the test set is what is really causing the pain. I'm waiting to see the reply by the OP though. This is an interesting problem (I am dealing with a similar one myself right now). $\endgroup$ – darXider Mar 1 '17 at 21:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.