1
$\begingroup$

EDIT: Earlier this question got closed because my question was not precise enough and really contained several questions. I have now tried to make the question more precise. I hope it's ok now.


I have the following (already scaled and centered) data set:

# A tibble: 55,166 x 17
   Target    TotalOrders   TotalSpending      Spending_A   Spending_B     Spending_C
   <fct>         <dbl>         <dbl>             <dbl>      <dbl>           <dbl>
 1 0             2             0.0180            0          0.0186          0      
 2 1             1             0.0282            0          0               0      
 3 1             1             0.0161            0          0               0      
 4 1             1             0.0332            0.124      0               0      
 5 1             2.13          0.0240            0          0               0.00502
 6 0             1             0.00739           0          0               0      
 7 1             1             0.0101            0          0               0.0277 
 8 0             1             0.0185            0          0.0191          0      
 9 0             1             0.0362            0          0.0368          0      
10 0             1             0.0135            0.0519     0               0      
# ... with 55,156 more rows, and 11 more variables: Spending_D <dbl>,
#   Spending_E <dbl>, Spending_F <dbl>, Spending_G <dbl>,
#   Quantity_A <dbl>, Quantity_B <dbl>, Quantity_C <dbl>, Quantity_D <dbl>,
#   Quantity_E <dbl>, Quantity_F <dbl>, Quantity_G <dbl>

Each line refers to one unique customer. Explanation of variables:

  • Target: 1 if customer placed an order, 0 if customer did not.
  • TotalOrders: Number of orders a customer has placed (scaled).
  • TotalSpending: Total amount of money a customer spent (scaled).
  • Spending_X: How much customer spent in product category X (scaled).
  • Quantity_X: How many orders customer placed in product category X (scaled).

I want to predict the probability of a customer placing an order and I thought logistic LASSO regression would be a good idea for this purpose since it predicts probabilities. Here is what I did:

Split data into test and train

df_split <- initial_split(df, strata = Target) 
df_train <- training(df_split) 
df_test <- testing(df_split) 

# sample to reshuffle data set

df_train <- sample_n(df_train, nrow(df_train))
x.train <- df_train[,2:ncol(df_train)]
y.train <- df_train[,1]

df_test <- sample_n(df_test, nrow(df_test)) 
x.test <- df_test[,2:ncol(df_test)]
y.test <- df_test[,1]

Use cross-validation to find optimal lambda

OBS: note the bad misclassification rate!

library(glmnet)
library(caret)

cross_val <- cv.glmnet(as.matrix(x.train), as.matrix(y.train), 
                       family = 'binomial', 
                       type.measure = 'class',
                       alpha = 1, 
                       nlambda = 100)

enter image description here

Fit model using lambda_1se and lambda_min

fit_1se <- glmnet(as.matrix(x.train), as.matrix(y.train), 
              family = 'binomial', 
              alpha = 1, 
              lambda = cross_val$lambda.1se)

fit_min <- glmnet(as.matrix(x.train), as.matrix(y.train), 
              family = 'binomial', 
              alpha = 1, 
              lambda = cross_val$lambda.min)

Make predictions on the test set

predictions_1se <- predict(fit_1se, newx = as.matrix(x.test), type = 'response')
predictions_min <- predict(fit_min, newx = as.matrix(x.test), type = 'response')

What I need is the predictions to be 0 or 1 only so that I can calculate the missclassification rate and accuracy using y.test, but printing a prediction i get:

73712   0.4832631
10180   0.5026904
68423   0.4833344
177921  0.4616881
32901   0.4835719
15900   0.4483584
32876   0.5015573
67358   0.5009118
24543   0.6169297
6814    0.5108814
64806   0.4842850
74783   0.5038235
78119   0.4832987
[ reached getOption("max.print") -- omitted 17390 rows ]

Question:

It feels like I'm using a bad procedure for this data set and the task at hand because I'm getting all my predictions to be close to 0.5 so it's not any better than randomly guessing. I'm quite new in this field and want to know whether my procedure above is ok and what scoring-rules I should use.

In summary: based on the data provided above, I want to predict the probabilities for the target variable but I'm not sure about my own way of going about it. How can I verify that my predictions are correct/not correct?

$\endgroup$
6
  • 1
    $\begingroup$ I dispute your statement that "What I need is the predictions to be 0 or 1 only so that I can calculate the missclassification rate and accuracy". Misclassification and accuracy are highly misleading. Why is accuracy not the best measure for assessing classification models? Instead, assess your probabilistic classifications using proper scoring rules. $\endgroup$ Nov 16 '21 at 15:01
  • 1
    $\begingroup$ As to which threshold to use, blindly using 0.5 is very limiting. The threshold to use to map your probabilistic classifications to actions/decisions (!) needs to take costs into account, not only the classifications. Classification probability threshold $\endgroup$ Nov 16 '21 at 15:02
  • $\begingroup$ Hello @StephanKolassa, thanks for your comments! I've read both of your links. In your answer in the first link you state that one has to take into account the consequences of the decision when determening metric. However in my case there are no grave consequences except that for any customer who has the probability of purchase $> 0.5$ we send a certain email and if $\mathbb{P} \le 0.5$ we send another. So all I need are the probabilities and I want to know if my method above is a correct way of doing this sort of task. PS: I've updated my question aswell, I hope it's clearer now what I want. $\endgroup$
    – Parseval
    Nov 16 '21 at 20:56
  • $\begingroup$ @StephanKolassa - You also link to "proper scoring rules". Maybe I've not looked hard enough but when I click on the link I can't seem to find any similar situation nor any description of how to implement any such rule. I found this: stats.stackexchange.com/questions/536317/…, but there you state that he needs error classification instead. Apologies for the link like that but I don't know how to hyperlink in comments. $\endgroup$
    – Parseval
    Nov 16 '21 at 21:10
  • $\begingroup$ I have posted an answer, hope it helps. Note that the thread you link in your last comment looks at predicting a continuous outcome, where things are slightly different than in your discrete setting (and your case is arguably simpler). $\endgroup$ Nov 17 '21 at 7:29
2
$\begingroup$

Your code looks technically correct to me (it's hard to give a deeper answer without a reproducible example). Per my comments, I do not believe type.measure = 'class' is appropriate; default or mse would be a better choice.

It is always hard to say whether your ML algorithm has detected all the structure there is in your data: How to know that your machine learning problem is hopeless? There may be nothing more to find here, so the best prediction is essentially guessing - or there may be something left. It's always good to talk to the domain experts. Perhaps you are already sending out (personalized?) reminders, coupons or promotions - can you include these in your model?

The typical reaction is to use a more complex model. I do not subscribe to this automatic response, but you do have a decent-sized data set, so you might want to look at models that provide a little more flexibility. For instance, there may be interactions between your predictors, and a logistic regression will not capture these unless they are explicitly modeled. There are simple implementations of GBMs or Random Forests out there, and if interpretability is not high on your list of requirements, you can probably easily modify your code to try one of these models.

In a comment, you ask how to apply proper scoring rules to your problem. Note that your logistic regression (with type = 'response') outputs predicted probabilities $\hat{p}_i$ of the $i$-th instance to belong to the target class, i.e., to lead to a purchase. So you conversely have a predicted probability $1-\hat{p}_i$ that the $i$-th instance is not a purchase. In your test set, you know whether it was or was not a purchase - so any prediction with a high $\hat{p}_i$ for a purchase (and a low $\hat{p}_i$ for a non-purchase) is a good one. The most common proper scoring rules are the Brier score $$ B = -\sum_{i\text{ purchase}} \hat{p}_i^2 - \sum_{i\text{ non-purchase}} (1-\hat{p}_i)^2$$ and the log score $$ L = -\sum_{i\text{ purchase}} \log \hat{p}_i - \sum_{i\text{ non-purchase}} \log(1-\hat{p}_i).$$ In both cases, lower is better. (There is also the opposite convention, dropping the minus signs; then higher is better.) The tag wiki contains more information and pointers to literature. Of particular interest may be Why is LogLoss preferred over other proper scoring rules?

Incidentally, here is a problem: if you take decisions based on your model output (sending one of two different types of email) and then use the result to evaluate your model, you have introduced a confounder. This problem of course applies regardless of what evaluation measure you use. There is no remedy that completely satisfies me, but it is good to keep this complication in mind. See How to avoid selection bias while updating lead scoring (predictive) model with new data.

$\endgroup$
4
  • $\begingroup$ Wonderful answer as always Stephan, thank you! However, the Brier score against what should I measure it? For example, if $B=-9.9$, how do I know that this "Good" or "low enough to be good"? Sure if I know that the minimum is -10, then it's comparably close to -10. However if the minimum can be whatever then I have no reference. I'm probably misunderstanding the Breier score. $\endgroup$
    – Parseval
    Nov 18 '21 at 8:43
  • $\begingroup$ Also, according to wikipedia, I should divide the sum by the number of probabilities, (upper limit of the sum). $\endgroup$
    – Parseval
    Nov 18 '21 at 8:50
  • $\begingroup$ First, yes, you typically divide by the number of instances, you are right about that - I left this off because it doesn't make a difference if you are using the scores to decide between different models applied to the same test dataset, and to keep the formulas simpler. $\endgroup$ Nov 18 '21 at 12:40
  • 1
    $\begingroup$ Second, what is a "good" or "acceptable" score is always hard to say - exactly as it is hard to say what is a "good" or "acceptable" accuracy, MSE or MAPE. An accuracy of 99% sounds good, unless you are running a nuclear reactor, and a misclassification of some condition could lead to a major meltdown. It always depends on what you use your model outputs for later on. Per that "hopeless" thread, there is usually also no "optimal" score you could aim for, because we are usually not in a coin tossing context where we know that there is no way to get better than 0.5 accuracy. $\endgroup$ Nov 18 '21 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.