I am trying to estimate the probability for a user to get to a specific page in a website. The collected information for the users are
- whether they use a PC or a mobile device
- referral page
All the variables are categorical and I have two possible outcome, either success (1) or not success (0).
I thought of using logistic regression as my dependent variable is categorical. I would get the probabilities for the two classes using the
predict_proba method in sklearn implementation of the classifier. However, I have some doubts:
- The classes are not perfectly balanced, i.e. $\sim$1 in 10 users has outcome 1. I compared my model with a dummy classifier and they equally perform;
- performing a grid search over the classifier, any combination of values leads to recall of zero. That is not only does the classifier outperform the dummy one, but does it not find the success-labeled users.
Given an intelligent threshold as said in this similar question, could I use the prediction to assess such a success probability for a user? And, given that I have 3 categorical variables, should I think of the observations/variables ratio before or after the one-hot encoding process?