1
$\begingroup$

Lets say you want to create a random forest model that predicts whether a user will click on your search engine ad. Lets say the training data set marks the independent variable (i.e. whether the user clicked or not) as a 0 if it's a miss, and a 1 if it's a hit.

Should this type of model be a classifier or a regression model? Or can it be either?

Background

I am trying this both ways, and I am finding that in the classifier version of this model, the model predicts a miss nearly every time. I'm not sure if there is a problem in the way I've defined the model, or if this is by design, because for a given data point, if you repeated that data point 100 times it might only be a hit 10% of the time, even for the likely-to-be-a-hit data points.

In other words, there is almost no single point where the user is actually likely to click on the ad, so I think that maybe given that the probability is always less than 0.5, the model is simply predicting a miss every time

$\endgroup$
6
  • 3
    $\begingroup$ It sounds like you have unbalanced traning data set. Can you provide frequency of the miss and hit classes in training data? $\endgroup$ Dec 3 '13 at 20:53
  • $\begingroup$ The training set is balanced. The problem is that the hit rate is so infrequent that a classifier would always predict a miss if forced to make a binary decision. If your random-forest classifier can also output probabilities in the prediction, it's much more meaningful. $\endgroup$
    – user35581
    Dec 4 '13 at 13:40
  • $\begingroup$ I concur with @Vojtech R. this is probably due to an imbalanced target variable (e.g. 10% 1's and 90% 0's) How many cases do you have of each (1's and 0's)? Another thing that could be happening is that your independent variables don't explain your target very well. $\endgroup$
    – JEquihua
    Dec 4 '13 at 16:04
  • $\begingroup$ ~1000 datapoints, and ~100 1s. Maybe I wasn't understanding your definition of unbalanced, but if you're saying that there are far more 0s than 1s, then yes, there are, but there is still enough data for a random forest classifier model. The data set is unbalanced by design, as the results will be unbalanced. $\endgroup$
    – user35581
    Dec 4 '13 at 16:51
  • $\begingroup$ Also, the probability of a hit should never really exceed ~0.3, which would still be a probable miss, but that's ok. $\endgroup$
    – user35581
    Dec 4 '13 at 16:58
1
$\begingroup$

So what I can gather from your response you have a unbalanced data set with 100 1's and 900 0's. Since random forests are built on Classification and Regression Trees (CART) they are highly sensitive to unbalanced data. Let me explain a bit more, CARTs work by means of a divide a conquer approach. This means that they partition your feature space into subsets where your target variable behaves more homogeneously. For example, if you are looking at income as a target, maybe partitioning the population into people under 16 years old and people of 17 and over helps you obtain two subsets where income behaves in a more constant fashion. You can do this more thoroughly and with more variables until you have quite small subsets where a simple average helps you explain the behavior of the subset. In the classification scheme you replace an average with a majority vote, meaning if you introduce a new observation into the tree you just trained, it will follow certain rules (e.g. is older than 17 years old, is a female, is living in Milan) until it ends up in a certain subset with, for example, 6 observations labeled "high income" and 1 observation labeled "low income", 6 > 1; so you label it "high income" . If you have 100 1's and 900 0's then it is quite possible that every single subset created has more 0's than 1's so you will always classify things as 0.

There are many techniques to tackle unbalanced data sets, luckily for you random forest has some quite straight forward approaches to do this. Note that a random forest is an ensemble of orthogonal trees, as en ensemble EACH tree will cast a vote and the majority of that will win (a majority vote of majority votes). Now, each tree is built on bootstrapped sample of your whole data. You may force the bootstrapped samples to be balanced themselves. To do this in R you would simply choose the sampsize in the random forest implementation accordingly:

binaryclassifier <-randomForest(y=as.factor(yourtarget),
                                x=independent_variable_matrix,ntrees=2000,
                                sampsize=c(60,60)) # so you would end up with
                                                  # each tree grown from a 120
                                                  # observation, balanced data set

# To check out how precise this is you can simply look at this plot
plot(binaryclassifier)

# Or check out the confusion matrix
binaryclassifier$confusion

These are based on out of bag samples. When you construct each tree you take a bootstrap sample which in turn means you left some of the observations out of that trees construction. These are called "out of bag" samples and can be used to test the model on the fly. You can also play around a bit with the sample size, a perfectly balanced data sample is not always what works best.

Another thing that can work for you but is a bit more advanced is to use cross validation to find an optimal threshold for classifying new observations into 1's or 0's.

Elaborating a bit you have a ratio of 0.1 1's on your whole data set. You could train a random forest as you have been doing now where all the predictions give you 0. But instead of taking the default random forest prediction (majority vote) you could obtain the probabilities:

predictions <- predict(binaryclassifier, newdata, type="prob")

# look at the predcitions structure
head(predictions)

And manually set all the observations which have a ratio of 1's to 0's larger than 0.1 to be labeled as 1.

This 0.1 is just a guess, as I mentioned, you could find an optimal threshold through cross validation. Good luck!

$\endgroup$
4
  • $\begingroup$ Thanks - this helps a lot. Is the default randomforest sample size the entire data set? When I compare my model against my test data set, 60 + 60 seems to be a good start, and when I use 10 + 10 it produces even better results. I don't have a good grasp on how I choose the sample sizes, although the trial and error approach seems to kind of work. $\endgroup$
    – user35581
    Dec 5 '13 at 16:26
  • $\begingroup$ the default takes a sample of size 0.632 of your whole data set (rounded up) for each tree of the ensemble, since it is a bootstrap sample you would end up with a proportion similar to the whole: 0.1 1's and 0.9 0's. It's a tad strange that you get better results with (10,10). It's a tiny sample! Usually a little bit more is better. I guess the numbers speak for themselves though. $\endgroup$
    – JEquihua
    Dec 5 '13 at 16:37
  • $\begingroup$ Yeah, I agree. I am thinking that by sampling a small set maybe individual trees get really good at picking out a particular set of traits that is predictive. Maybe there are 10 different profiles of a user who clicks on an ad, and by creating small subsets, each of those profiles gets trained on (randomly) whereas in a larger set of 100's of samples, those profiles would all kind of average out together and become more meaningless... I'm not sure if my theory has any validity, I'm just trying to brainstorm why such a small set might actually be useful... $\endgroup$
    – user35581
    Dec 5 '13 at 16:59
  • $\begingroup$ I really have no idea. Maybe it is related to this question I answered a while ago: stats.stackexchange.com/questions/77904/… $\endgroup$
    – JEquihua
    Dec 5 '13 at 17:23
1
$\begingroup$

Your problem statement is as close to the definition of binary classification as one can get.

It's worth observing that many popular classification algorithms aren't binary internally. Decision trees, neural nets, SVM, ... all have decision values which you could use to calculate a probability.

Additionally, if your classifier always predicts miss, something is likely wrong. Try assigning higher weights to positive training instances in whichever way the algorithm of your choosing allows this.

$\endgroup$
0
$\begingroup$

I believe this should be a classifier problem after discussing it with my colleague.

A classifier RandomForest model can also output probabilities (at least in R). In the R predict method for random-forest, you need to set type="prob" for example:

predict(myRandomForest, newData, type="prob")
$\endgroup$
0
$\begingroup$

You have too few positive events, so there is very little meaningful information for your classifier to actually figure out what factors to seperate clicking on an ad and not

It's worthwhile to do some descriptive statistics. You have some attributes that you are looking at to build your random forest, why not to evaluate the distribution of those attributes between the "clicked" and "not-clicked" ads? You can do a chi-squared test to figure out if any of those attributes are actually informative, the truth is none may be, or only a handful are and everything else is not relevant and should be removed.

The next thing that may be interesting to try is to create an artificial training set, which would balance clicks and the non-clicks. Because you have so few positives you may want to consider sampling with replacement. So, imagine building a training set of size 400, with 200 ads that were clicked and 200 ads that were not clicked. It may build a better classifier than your current training set. I would bootstrap this to get a sense of how the selection of a training set impacts your classifier performance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.