5
$\begingroup$

I have a dataset which has Quantity ordered (along with other variables like product type, product price, customer group etc). Target variable is whether customer churned or not. I am doing this to convert my continuous variable into categorical values like high, med, low based on Qty ordered level

However, my question is not based on the dataset itself but on the technique called supervised binning.

Doesn't supervised binning qualify as data leakage? because we create bins based on the target variable (train data only). Later, we use that info (bin info based on target column) and feed it as input to the model.

Can you share some insights on whether it is recommended to this?

If yes, why so?

If not, why so? Because, I see lot of tutorials and posts on doing supervised binning for discretization of continuous variables (during data preparation). Should I only use unsupervised binning?

$\endgroup$
10
  • $\begingroup$ Why do you want to discretize your data in the first place? $\endgroup$ Commented Feb 10, 2022 at 7:24
  • $\begingroup$ I am trying to bin to gain some insights. Meaning, if I know that orders with qty values between 3400 to 6500 leads to churn more often, then thats useful. If I have them as continuous, it may not give such insights (rather than letting me know that Qty is an important variable). Instead of me manually creating bins, I want the system to find out cut points automatically $\endgroup$
    – The Great
    Commented Feb 10, 2022 at 7:30
  • $\begingroup$ I also felt discretization helps handle outliers (putting them in one bucket) $\endgroup$
    – The Great
    Commented Feb 10, 2022 at 7:32
  • 3
    $\begingroup$ Discretization loses a lot of information, stats.stackexchange.com/q/68834/1352, so I would argue it will rather reduce insight than create it. It will probably be much better to use your quantity as a numerical variable, possibly spline transform it, and derive your insights from plots of model fits or predictions. Also, yes, binning will leak data, unless you bin based on the training data only. $\endgroup$ Commented Feb 10, 2022 at 7:37
  • $\begingroup$ Is spline transform better over quartile/decile based discretization? Would you suggest unsupervised discretization? $\endgroup$
    – The Great
    Commented Feb 10, 2022 at 7:50

2 Answers 2

7
$\begingroup$

As already noticed in the comments and another answer, you need to train the binning algorithm using training data only, in such a case it has no chance to leak the test data, as it hasn't seen it.

But you seem to be concerned with the fact that the binning algorithm uses the labels, so it "leaks" the labels to the features. This concern makes sense, in the end if you had a model like

$$ y = f(y) $$

it would be quite useless. It would predict nothing and it would be unusable at prediction time, when you have no access to the labels. But it is not that bad.

First, notice that any machine learning algorithm has access to both labels and the features during the training, so if you weren't allowed to look at the labels while training, you couldn't do it. The best example would be naive Bayes algorithm that groups the data by the labels $Y$, calculates the empirical probabilities for the labels $p(Y=c)$, and the empirical probabilities for the features given (grouped by) each label $p(X_i | Y=c)$, and combines those using Bayes theorem

$$ p(Y=c) \prod_{i=1}^n p(X_i | Y=c) $$

If you think about it, it is almost like a generalization of the binning idea to the smooth categories: in binning we transform $X_i | Y=c$ to discrete bins, while naive Bayes replaces it with a probability (continous score!). Of course, the difference is that with binning you then use the features as input for another model, but basically the idea is like a kind of poor man's naive Bayes algorithm.

Finally, as noticed by Stephan Kolassa in the comment, binning is usually discouraged. It results in loosing information, so you have worse quality features to train as compared to the raw data. Ask yourself if you really need to bin the data in the first place.

$\endgroup$
0
2
$\begingroup$

If you only use training data for supervised binning, you cannot leak information from the test dataset, simply because you are not using it. So, no, when done right, there is no leakage.

$\endgroup$
5
  • $\begingroup$ Sorry, can I check how can supervised binning be done without target/label data? $\endgroup$
    – The Great
    Commented Feb 10, 2022 at 7:50
  • $\begingroup$ You mean, I can use train data and train labels, to create bins? Is that what you are suggesting? But I should not use the full datsaet to do the binning? btw, upvoted for the help $\endgroup$
    – The Great
    Commented Feb 10, 2022 at 7:51
  • $\begingroup$ Yes, that's what I mean. That's what you always do in supervised learning: you use both the input and target data for training. Leakage concerns whether you somehow sneak information from the test dataset into your training algorithms, so that the algorithm already knows about the test dataset and thus performs better. $\endgroup$
    – frank
    Commented Feb 10, 2022 at 7:55
  • $\begingroup$ But my question on data leakage was not based on test info slipping into train, but the target info (from train) used as bin info (created as input variable during train). Doesn;t this qualify as data leakage as well? This sort of leakage can lead to overfitting? No? $\endgroup$
    – The Great
    Commented Feb 10, 2022 at 8:01
  • 2
    $\begingroup$ Using information from the target part of your training data to bin the input part of your training data does not qualify as leakage. You are free to use any part of your training data in any way you wish. Whether some ways lead to more overfitting than others is a different matter, it doesn't have anything to do with leakage. $\endgroup$
    – frank
    Commented Feb 10, 2022 at 8:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.