4
$\begingroup$

I am trying to estimate the probability for a user to get to a specific page in a website. The collected information for the users are

  • countries
  • whether they use a PC or a mobile device
  • referral page

All the variables are categorical and I have two possible outcome, either success (1) or not success (0).

I thought of using logistic regression as my dependent variable is categorical. I would get the probabilities for the two classes using the predict_proba method in sklearn implementation of the classifier. However, I have some doubts:

  1. The classes are not perfectly balanced, i.e. $\sim$1 in 10 users has outcome 1. I compared my model with a dummy classifier and they equally perform;
  2. performing a grid search over the classifier, any combination of values leads to recall of zero. That is not only does the classifier outperform the dummy one, but does it not find the success-labeled users.

Given an intelligent threshold as said in this similar question, could I use the prediction to assess such a success probability for a user? And, given that I have 3 categorical variables, should I think of the observations/variables ratio before or after the one-hot encoding process?

Improve this question
$\endgroup$
3
  • 5
    $\begingroup$ Generally speaking, and according to [1], using an imbalance dataset may have an effect on the intercept coefficient value and, consequently, the estimated probability. [1] King, G. and Zeng, L. (2001). Logistic regression in rare events data. Political Analysis, 9(2):137–163. $\endgroup$
    – PatternRecognition
    Commented Jan 5, 2018 at 14:56
  • 4
    $\begingroup$ A word on informal terminology. Highly imbalanced usually means something like 0.01% of one class, i.e. severe rarity of an event. 10% prevalence is well within the domain where no special methods are needed to account for the class rarity. The ML community has a weird obsession with perfectly balanced datasets, but it's worth casting a skeptical eye towards. Good on you for going with predict_proba. $\endgroup$
    – Matthew Drury
    Commented Jan 8, 2018 at 15:22
  • $\begingroup$ @MatthewDrury thanks for the explanation, I edited the question and removed the tag. $\endgroup$
    – Mattia Paterna
    Commented Jan 8, 2018 at 15:32

1 Answer 1

Reset to default
5
$\begingroup$

Don't use a threshold at all. A logistic regression already outputs a probabilistic prediction - work directly with that. Assess whether it is good using proper scoring rules. Here is something I wrote about thresholds earlier.

Whether your dataset is unbalanced or not won't matter. (Except that you will need a larger sample size than for balanced data.)

That your recall is zero and no better than a dummy classifier's may simply mean that there is very little structure to your problem. Simple models are often surprisingly hard to beat. Related: Why is accuracy not the best measure for assessing classification models?

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Thanks for your answer Stephan, I really appreciate the related links. The point here is that, given my model, I will never get a probability of success $p > 0.5$ - and I agree with it. Therefore I was interested in directly giving such the probabilistic prediction but I did not know if it was completely correct. $\endgroup$
    – Mattia Paterna
    Commented Jan 8, 2018 at 8:22
  • 1
    $\begingroup$ Agreed, one remark, I've seen this in practice a lot. Performance may not be better than a dummy classifier because no predictions end up above 50%. Computing accuracy or recall is then so coarse that you may well have a good classifier, but it's not measured by these measures. To your point, actually. My recommendation is to use logloss instead. $\endgroup$
    – Gijs
    Commented Jan 8, 2018 at 13:20
  • $\begingroup$ @Gijs thanks for you comment - do you mean I should use the log loss as scoring rule? $\endgroup$
    – Mattia Paterna
    Commented Jan 8, 2018 at 15:08
  • 1
    $\begingroup$ @MattiaPaterna, yes, see scikit-learn.org/stable/modules/model_evaluation.html#log-loss! $\endgroup$
    – Gijs
    Commented Jan 8, 2018 at 15:28
  • 2
    $\begingroup$ Log loss is one idea. It's a proper scoring rule (the so-called logarithmic score). Alternatively, you could look at the Brier score. That you don't get a classification probability higher than 50% can certainly happen and be completely correct, especially in the context of rare-but-not-overly-so events, like consumer credit defaults. $\endgroup$
    – Stephan Kolassa
    Commented Jan 8, 2018 at 15:43

Not the answer you're looking for? Browse other questions tagged or ask your own question.