4
$\begingroup$

This question already has an answer here:

I am trying to estimate the probability for a user to get to a specific page in a website. The collected information for the users are

  • countries
  • whether they use a PC or a mobile device
  • referral page

All the variables are categorical and I have two possible outcome, either success (1) or not success (0).

I thought of using logistic regression as my dependent variable is categorical. I would get the probabilities for the two classes using the predict_proba method in sklearn implementation of the classifier. However, I have some doubts:

  1. The classes are not perfectly balanced, i.e. $\sim$1 in 10 users has outcome 1. I compared my model with a dummy classifier and they equally perform;
  2. performing a grid search over the classifier, any combination of values leads to recall of zero. That is not only does the classifier outperform the dummy one, but does it not find the success-labeled users.

Given an intelligent threshold as said in this similar question, could I use the prediction to assess such a success probability for a user? And, given that I have 3 categorical variables, should I think of the observations/variables ratio before or after the one-hot encoding process?

$\endgroup$

marked as duplicate by kjetil b halvorsen, Peter Flom Jul 3 '18 at 21:40

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 5
    $\begingroup$ Generally speaking, and according to [1], using an imbalance dataset may have an effect on the intercept coefficient value and, consequently, the estimated probability. [1] King, G. and Zeng, L. (2001). Logistic regression in rare events data. Political Analysis, 9(2):137–163. $\endgroup$ – PatternRecognition Jan 5 '18 at 14:56
  • 4
    $\begingroup$ A word on informal terminology. Highly imbalanced usually means something like 0.01% of one class, i.e. severe rarity of an event. 10% prevalence is well within the domain where no special methods are needed to account for the class rarity. The ML community has a weird obsession with perfectly balanced datasets, but it's worth casting a skeptical eye towards. Good on you for going with predict_proba. $\endgroup$ – Matthew Drury Jan 8 '18 at 15:22
  • $\begingroup$ @MatthewDrury thanks for the explanation, I edited the question and removed the tag. $\endgroup$ – Mattia Paterna Jan 8 '18 at 15:32
5
$\begingroup$

Don't use a threshold at all. A logistic regression already outputs a probabilistic prediction - work directly with that. Assess whether it is good using proper scoring rules. Here is something I wrote about thresholds earlier.

Whether your dataset is unbalanced or not won't matter. (Except that you will need a larger sample size than for balanced data.)

That your recall is zero and no better than a dummy classifier's may simply mean that there is very little structure to your problem. Simple models are often surprisingly hard to beat. Related: Why is accuracy not the best measure for assessing classification models?

$\endgroup$
  • 1
    $\begingroup$ Thanks for your answer Stephan, I really appreciate the related links. The point here is that, given my model, I will never get a probability of success $p > 0.5$ - and I agree with it. Therefore I was interested in directly giving such the probabilistic prediction but I did not know if it was completely correct. $\endgroup$ – Mattia Paterna Jan 8 '18 at 8:22
  • 1
    $\begingroup$ Agreed, one remark, I've seen this in practice a lot. Performance may not be better than a dummy classifier because no predictions end up above 50%. Computing accuracy or recall is then so coarse that you may well have a good classifier, but it's not measured by these measures. To your point, actually. My recommendation is to use logloss instead. $\endgroup$ – Gijs Jan 8 '18 at 13:20
  • $\begingroup$ @Gijs thanks for you comment - do you mean I should use the log loss as scoring rule? $\endgroup$ – Mattia Paterna Jan 8 '18 at 15:08
  • 1
    $\begingroup$ @MattiaPaterna, yes, see scikit-learn.org/stable/modules/model_evaluation.html#log-loss! $\endgroup$ – Gijs Jan 8 '18 at 15:28
  • 2
    $\begingroup$ Log loss is one idea. It's a proper scoring rule (the so-called logarithmic score). Alternatively, you could look at the Brier score. That you don't get a classification probability higher than 50% can certainly happen and be completely correct, especially in the context of rare-but-not-overly-so events, like consumer credit defaults. $\endgroup$ – Stephan Kolassa Jan 8 '18 at 15:43

Not the answer you're looking for? Browse other questions tagged or ask your own question.