Logistic LASSO regression model in R (glmnet) - predictions very close to 0.5 and bad misclassification error

Question

EDIT: Earlier this question got closed because my question was not precise enough and really contained several questions. I have now tried to make the question more precise. I hope it's ok now.

---

I have the following (already scaled and centered) data set:

# A tibble: 55,166 x 17
   Target    TotalOrders   TotalSpending      Spending_A   Spending_B     Spending_C
   <fct>         <dbl>         <dbl>             <dbl>      <dbl>           <dbl>
 1 0             2             0.0180            0          0.0186          0      
 2 1             1             0.0282            0          0               0      
 3 1             1             0.0161            0          0               0      
 4 1             1             0.0332            0.124      0               0      
 5 1             2.13          0.0240            0          0               0.00502
 6 0             1             0.00739           0          0               0      
 7 1             1             0.0101            0          0               0.0277 
 8 0             1             0.0185            0          0.0191          0      
 9 0             1             0.0362            0          0.0368          0      
10 0             1             0.0135            0.0519     0               0      
# ... with 55,156 more rows, and 11 more variables: Spending_D <dbl>,
#   Spending_E <dbl>, Spending_F <dbl>, Spending_G <dbl>,
#   Quantity_A <dbl>, Quantity_B <dbl>, Quantity_C <dbl>, Quantity_D <dbl>,
#   Quantity_E <dbl>, Quantity_F <dbl>, Quantity_G <dbl>

Each line refers to one unique customer. Explanation of variables:

• Target: 1 if customer placed an order, 0 if customer did not.
• TotalOrders: Number of orders a customer has placed (scaled).
• TotalSpending: Total amount of money a customer spent (scaled).
• Spending_X: How much customer spent in product category X (scaled).
• Quantity_X: How many orders customer placed in product category X (scaled).

I want to predict the probability of a customer placing an order and I thought logistic LASSO regression would be a good idea for this purpose since it predicts probabilities. Here is what I did:

Split data into test and train

df_split <- initial_split(df, strata = Target) 
df_train <- training(df_split) 
df_test <- testing(df_split) 

# sample to reshuffle data set

df_train <- sample_n(df_train, nrow(df_train))
x.train <- df_train[,2:ncol(df_train)]
y.train <- df_train[,1]

df_test <- sample_n(df_test, nrow(df_test)) 
x.test <- df_test[,2:ncol(df_test)]
y.test <- df_test[,1]

Use cross-validation to find optimal lambda

OBS: note the bad misclassification rate!

library(glmnet)
library(caret)

cross_val <- cv.glmnet(as.matrix(x.train), as.matrix(y.train), 
                       family = 'binomial', 
                       type.measure = 'class',
                       alpha = 1, 
                       nlambda = 100)

Fit model using lambda_1se and lambda_min

fit_1se <- glmnet(as.matrix(x.train), as.matrix(y.train), 
              family = 'binomial', 
              alpha = 1, 
              lambda = cross_val$lambda.1se)

fit_min <- glmnet(as.matrix(x.train), as.matrix(y.train), 
              family = 'binomial', 
              alpha = 1, 
              lambda = cross_val$lambda.min)

Make predictions on the test set

predictions_1se <- predict(fit_1se, newx = as.matrix(x.test), type = 'response')
predictions_min <- predict(fit_min, newx = as.matrix(x.test), type = 'response')

What I need is the predictions to be 0 or 1 only so that I can calculate the missclassification rate and accuracy using y.test, but printing a prediction i get:

73712   0.4832631
10180   0.5026904
68423   0.4833344
177921  0.4616881
32901   0.4835719
15900   0.4483584
32876   0.5015573
67358   0.5009118
24543   0.6169297
6814    0.5108814
64806   0.4842850
74783   0.5038235
78119   0.4832987
[ reached getOption("max.print") -- omitted 17390 rows ]

Question:

It feels like I'm using a bad procedure for this data set and the task at hand because I'm getting all my predictions to be close to 0.5 so it's not any better than randomly guessing. I'm quite new in this field and want to know whether my procedure above is ok and what scoring-rules I should use.

In summary: based on the data provided above, I want to predict the probabilities for the target variable but I'm not sure about my own way of going about it. How can I verify that my predictions are correct/not correct?

Answer 1

Your code looks technically correct to me (it's hard to give a deeper answer without a reproducible example). Per my comments, I do not believe type.measure = 'class' is appropriate; default or mse would be a better choice.

It is always hard to say whether your ML algorithm has detected all the structure there is in your data: How to know that your machine learning problem is hopeless? There may be nothing more to find here, so the best prediction is essentially guessing - or there may be something left. It's always good to talk to the domain experts. Perhaps you are already sending out (personalized?) reminders, coupons or promotions - can you include these in your model?

The typical reaction is to use a more complex model. I do not subscribe to this automatic response, but you do have a decent-sized data set, so you might want to look at models that provide a little more flexibility. For instance, there may be interactions between your predictors, and a logistic regression will not capture these unless they are explicitly modeled. There are simple implementations of GBMs or Random Forests out there, and if interpretability is not high on your list of requirements, you can probably easily modify your code to try one of these models.

In a comment, you ask how to apply proper scoring rules to your problem. Note that your logistic regression (with type = 'response') outputs predicted probabilities $\hat{p}_i$ of the $i$-th instance to belong to the target class, i.e., to lead to a purchase. So you conversely have a predicted probability $1-\hat{p}_i$ that the $i$-th instance is not a purchase. In your test set, you know whether it was or was not a purchase - so any prediction with a high $\hat{p}_i$ for a purchase (and a low $\hat{p}_i$ for a non-purchase) is a good one. The most common proper scoring rules are the Brier score $$ B = -\sum_{i\text{ purchase}} \hat{p}_i^2 - \sum_{i\text{ non-purchase}} (1-\hat{p}_i)^2$$ and the log score $$ L = -\sum_{i\text{ purchase}} \log \hat{p}_i - \sum_{i\text{ non-purchase}} \log(1-\hat{p}_i).$$ In both cases, lower is better. (There is also the opposite convention, dropping the minus signs; then higher is better.) The tag wiki contains more information and pointers to literature. Of particular interest may be Why is LogLoss preferred over other proper scoring rules?

Incidentally, here is a problem: if you take decisions based on your model output (sending one of two different types of email) and then use the result to evaluate your model, you have introduced a confounder. This problem of course applies regardless of what evaluation measure you use. There is no remedy that completely satisfies me, but it is good to keep this complication in mind. See How to avoid selection bias while updating lead scoring (predictive) model with new data.