2
$\begingroup$

I'm building a predictive logistic regression model on whether someone books an Airbnb and the independent variables are things like how close the Airbnb is to the center of the city, it's ratings and price.

I've still very new at statistics and maybe I've just missed the answer to this, but I'm wondering if there's an issue when 90% of my entries for if they booked is 0 (failure) and 10% are 1 (success). Cause after splitting my data set into a training set (80% of data) and testing set (20% of data), it just seems like after I build my model and use predict function on my testing set with my training set model it just predicts everything to be 0 because there are just so many entries of that. So even though my classification rate is high I feel my model isn't really predicting anything with the independent variables and I'm only really learning anything about the independent variables from doing summary() of the model.

Any resources or suggestion to help my understanding would be appreciated.

$\endgroup$

2 Answers 2

1
$\begingroup$

Your intuition is correct. Your data can be described as "imbalanced," such that the distribution of the response variable is heavily skewed towards 0's. Therefore, it makes sense that your model only predicts 0's given the data you provided. There are a variety of sampling techniques that can be used to balance out the data without adding significant bias to the results. This article offers a nice explanation of the problem and some potential remedies, and this one offers sample R code for various strategies to overcome the data imbalance.

$\endgroup$
1
$\begingroup$

Many algorithms struggle with unbalanced classes, and can have their performance improved by artificially balancing the classes, but Logistic Regression isn't one of them.

In fact the only thing that is affected in a Logistic Regression by class imbalance is the intercept coefficient. For a 90/10 class imbalance, you should expect to see an intercept coefficient of about 2 (or -2, depending on your sign convention) assuming your other independent variables are centered. Every other term in the model can be interpreted as an odds ratio relative to that base-line probability of 0.1.

Logistic Regression is not called the Logistic Classifier after all - and this is not a nomenclature error. It really is a regression with the dependent variable being (a function of) the probability of a class conditioned on all the data you've given it.

While in extreme cases of imbalance, say probabilities of 1e-5, we can get into the realm of numeric instability because internal values used to represent such probability as close to what floating point numbers can accurately express, and then the gradients (used internally when fitting a model) are several orders of magnitude smaller than that, and the numeric error can accumulate. It's still a convex optimization problem and therefore reasonably robust against these kinds of problems, but when it starts to deal with extremes any algorithm is going to struggle. But my point is that your 90/10 imbalance is still very far from the realm where any of this starts to be a problem.

The bottom line is, if in the real world the chance of booking of only 10%, you can and should train logistic regression on a training set that has only 10% successes: it will return the favor by giving you the best unbiased probability estimates that it can for the model you've chosen.

You definitely should be using stratified sampling when you do your test/train split though. This ensures that class proportion of both is the same as the population class proportion.

Also, after your model is fitted and your using it to predict class labels, don't make the mistake of using the default threshold of 0.5. Remember: Logistic Regression emits probability estimates. It's entirely possible that given your data, the conditional probability of a booking never rises to 0.5 and the best your logistic regression model can ever do given the data that its seen is report a 0.2 or 0.3 chance of booking. This still represents a 2:1 or 3:1 lift over baseline, which may be a very strong model in certain business contexts.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.