Classifier or regression for binary system?

Question

Lets say you want to create a random forest model that predicts whether a user will click on your search engine ad. Lets say the training data set marks the independent variable (i.e. whether the user clicked or not) as a 0 if it's a miss, and a 1 if it's a hit.

Should this type of model be a classifier or a regression model? Or can it be either?

Background

I am trying this both ways, and I am finding that in the classifier version of this model, the model predicts a miss nearly every time. I'm not sure if there is a problem in the way I've defined the model, or if this is by design, because for a given data point, if you repeated that data point 100 times it might only be a hit 10% of the time, even for the likely-to-be-a-hit data points.

In other words, there is almost no single point where the user is actually likely to click on the ad, so I think that maybe given that the probability is always less than 0.5, the model is simply predicting a miss every time

Answer 1

So what I can gather from your response you have a unbalanced data set with 100 1's and 900 0's. Since random forests are built on Classification and Regression Trees (CART) they are highly sensitive to unbalanced data. Let me explain a bit more, CARTs work by means of a divide a conquer approach. This means that they partition your feature space into subsets where your target variable behaves more homogeneously. For example, if you are looking at income as a target, maybe partitioning the population into people under 16 years old and people of 17 and over helps you obtain two subsets where income behaves in a more constant fashion. You can do this more thoroughly and with more variables until you have quite small subsets where a simple average helps you explain the behavior of the subset. In the classification scheme you replace an average with a majority vote, meaning if you introduce a new observation into the tree you just trained, it will follow certain rules (e.g. is older than 17 years old, is a female, is living in Milan) until it ends up in a certain subset with, for example, 6 observations labeled "high income" and 1 observation labeled "low income", 6 > 1; so you label it "high income" . If you have 100 1's and 900 0's then it is quite possible that every single subset created has more 0's than 1's so you will always classify things as 0.

There are many techniques to tackle unbalanced data sets, luckily for you random forest has some quite straight forward approaches to do this. Note that a random forest is an ensemble of orthogonal trees, as en ensemble EACH tree will cast a vote and the majority of that will win (a majority vote of majority votes). Now, each tree is built on bootstrapped sample of your whole data. You may force the bootstrapped samples to be balanced themselves. To do this in R you would simply choose the sampsize in the random forest implementation accordingly:

binaryclassifier <-randomForest(y=as.factor(yourtarget),
                                x=independent_variable_matrix,ntrees=2000,
                                sampsize=c(60,60)) # so you would end up with
                                                  # each tree grown from a 120
                                                  # observation, balanced data set

# To check out how precise this is you can simply look at this plot
plot(binaryclassifier)

# Or check out the confusion matrix
binaryclassifier$confusion

These are based on out of bag samples. When you construct each tree you take a bootstrap sample which in turn means you left some of the observations out of that trees construction. These are called "out of bag" samples and can be used to test the model on the fly. You can also play around a bit with the sample size, a perfectly balanced data sample is not always what works best.

Another thing that can work for you but is a bit more advanced is to use cross validation to find an optimal threshold for classifying new observations into 1's or 0's.

Elaborating a bit you have a ratio of 0.1 1's on your whole data set. You could train a random forest as you have been doing now where all the predictions give you 0. But instead of taking the default random forest prediction (majority vote) you could obtain the probabilities:

predictions <- predict(binaryclassifier, newdata, type="prob")

# look at the predcitions structure
head(predictions)

And manually set all the observations which have a ratio of 1's to 0's larger than 0.1 to be labeled as 1.

This 0.1 is just a guess, as I mentioned, you could find an optimal threshold through cross validation. Good luck!

Answer 2

Your problem statement is as close to the definition of binary classification as one can get.

It's worth observing that many popular classification algorithms aren't binary internally. Decision trees, neural nets, SVM, ... all have decision values which you could use to calculate a probability.

Additionally, if your classifier always predicts miss, something is likely wrong. Try assigning higher weights to positive training instances in whichever way the algorithm of your choosing allows this.

Answer 3

I believe this should be a classifier problem after discussing it with my colleague.

A classifier RandomForest model can also output probabilities (at least in R). In the R predict method for random-forest, you need to set type="prob" for example:

predict(myRandomForest, newData, type="prob")

Answer 4

You have too few positive events, so there is very little meaningful information for your classifier to actually figure out what factors to seperate clicking on an ad and not

It's worthwhile to do some descriptive statistics. You have some attributes that you are looking at to build your random forest, why not to evaluate the distribution of those attributes between the "clicked" and "not-clicked" ads? You can do a chi-squared test to figure out if any of those attributes are actually informative, the truth is none may be, or only a handful are and everything else is not relevant and should be removed.

The next thing that may be interesting to try is to create an artificial training set, which would balance clicks and the non-clicks. Because you have so few positives you may want to consider sampling with replacement. So, imagine building a training set of size 400, with 200 ads that were clicked and 200 ads that were not clicked. It may build a better classifier than your current training set. I would bootstrap this to get a sense of how the selection of a training set impacts your classifier performance.