2
$\begingroup$

I already referred the posts here but this question is different. I don't wish to use categorical encoding. details given below

I have a dataset of 3000 unique customers purchase data. The dataset size is 380276.

I am working on a binary classification. The purchase data includes info on the below items

a) Name of the product (there are more than 1000 unique products)

b) segment to which the product belong to (there are 121 unique segments)

c) brokers who sold the product to customer (there are around 30 brokers)

d) Which country the customer belong to (there are more than 30 countries)

So, now I understand that one-hot encoding is not applicable (due to curse of dimensionality) and I don't wish to use hash-encoding (interpretability issue), label/ordinal encoding (imposes a rank order), target encoding (data leakage issue) etc.

So, I wish to transform high cardinal variable without using any encoding approach

In my dataset, a customer can have 300-400 transactions and they might have bought same product 400 times or 400 unique product (in 400 transactions).

Similarly, under a single product segment, there are multiple products. Of course, the level of cardinality of segment is less than products.

but what we think majorly drives the outcome is product. So, we wish to know whether a customer buying a specific product makes him a likely to stay with us or not. I don't know how can I reduce the cardinality of this product field

For ex: We have info on frequently purchased product by the customer. For instance, I know that customer A bought product A 60% of time and product B 20% of time and product C,D,E,F for 5% of the time. Not sure whether this qualifies as data leakage. Because, I am able to look back at the historical data now and compute the products that are bought most hy the customer.

Do you think knowing this, can help us reduce the cardinality of product variable for each customer? For ex: Can I create a column called Top product purchased for each customer? and Top 2 product purchased for each customer?

Can help me with the above?

Improve this question
$\endgroup$
8
  • 4
    $\begingroup$ there is no real problem with 1 hot encoding.the constrains of only selecting a single dimension at a time mean that there is little "curse of dimensionality". just use regularisation and add product hierarchies and other top level descriptors. typically the majority of sales happen for only a handful of products, so you only really have a much reduced feature set ( imagine if all your sales only came from 2 products. $\endgroup$
    – seanv507
    May 25, 2022 at 11:23
  • 1
    $\begingroup$ Why do you say there's a leakage problem with target encoding? If you fit/cross-validate with past-purchases predicting future ones, that would seem okay, right? In a similar manner random effects models (e.g. random effect logistic regression) and embedding layers in neural networks would seem okay, right? $\endgroup$
    – Björn
    May 25, 2022 at 11:29
  • $\begingroup$ @seanv507 - exactly. Most of our sales come from specific products (let's call it "special products" group ) So, do you suggest that I create a binary variable to indicate special product or not? btw, what do you mean by "add product hierarchies"? $\endgroup$
    – The Great
    May 25, 2022 at 11:36
  • $\begingroup$ @Björn - Using target label info (for training data) during training phase doesn't qualify as leakage? Because we get that target percentage from target label (which we may not have IRL).. Or you mean to say that I can use training data as and how I want with no concern about data leakage? $\endgroup$
    – The Great
    May 25, 2022 at 11:38
  • 4
    $\begingroup$ I am saying you can use 1 hot encoding of products with regularisation. in addition you could add product segment (what I meant by hierarchies). For those popular products the model will specialise specific to that product, but for less popular will use the product segment or other generic feature. $\endgroup$
    – seanv507
    May 25, 2022 at 12:05

2 Answers 2

Reset to default
2
$\begingroup$

What you're encountering is a pretty common issue.

Fortunately, you have a very good size dataset (380K rows), so even though curse of dimensionality applies, I think you might be able to get away with actually not putting that much effort into transforming your data. Have you tried just one-hot encoding it and seeing what happens? Also - did you mention one-hot encoding because you're using a neural network? Have you tried not one-hot encoding and using RandomForest?

This is not perfect, but I would recommend using your subject matter expertise (and knowledge of the data) to slightly reduce the dimensionality by grouping the data together a bit.

As a side note, you said "So, I wish to transform high cardinal variable without using any encoding approach". Someone should please correct me if I'm wrong, but I think that's impossible, by definition. If you're transforming it (esp. in a way that changes dimensionality or representation, you are by definition encoding it).

  1. Try just using the segment field instead of product (I know you said you think it's really the product, but just give it a try. See if the results you get are good enough). If not..

  2. Try using the product field as is in a RandomForest model (no encoding) and remove the segment field (it probably wouldn't add much value on top of the specific product, right?)

  3. Look at the frequency distribution of your product purchases. It'll probably be some kind of exponentially decaying distribution, where there are some products that account for hugely disproportionate fractions of all purchases (one product accounting for 5% of all purchases because it's so popular), and others that barely ever get purchased. Find what you think is an appropriate cutoff (for example, maybe you say if a product hasn't ever been purchased more than X times) you throw all of those into an "Other" bucket. This will probably be some combination of you trying different thresholds + reasonable deduction. If your dataset is 380K and you have a product purchased 5 or 10 times, your model probably won't be able to figure out anything useful about that specific product. So now, your "product" field becomes, for example, your top 100 products + one "Other" category that is everything else. See if this starts to give you any kind of result.

Improve this answer
$\endgroup$
2
$\begingroup$

So, now I understand that one-hot encoding is not applicable (due to curse of dimensionality)

That is not true. The curse of dimensionality is a problem for some algorithms, but it doesn't mean that you can't use data with many columns. You need to deal with it, for example, by using regularization. In many real-life data problems, you would have the number of columns as high or higher than in your case, for example in recommender systems (all users, all movies on Netflix) or natural language processing (all the words in the vocabulary). In cases like this, you have specialized models that deal with such data and technological solutions to deal with problems like high memory consumption (e.g. using sparse matrices). See One-hot-encoding gives untractable amount of classes for more details.

and I don't wish to use hash-encoding (interpretability issue),

Hashed representations are harder to interpret but not impossible to interpret. What it does is just randomly merge some categories to the same bins, so you have the OR relationship within the bin.

label/ordinal encoding (imposes a rank order),

Correct, it would not work here.

target encoding (data leakage issue) etc.

Data leakage would be a problem only if you implemented it incorrectly. If it always lead to leakage it won't be ever used, while it is used quite successively.

So, I wish to transform high cardinal variable without using any encoding approach.

You can't send the data to an algorithm "without representation". Every algorithm expects the data encoded as numbers (maybe except for tree-based models). Even for the algorithms that take as input "raw" data and do the feature engineering by themselves like neural networks you need to encode the data (images are numerical tensors, natural language data is words encoded somehow, etc). Even if instead of providing the data as one-hot columns, but "as is" to the algorithm, it would need to have something like separate weight per each category, or separate IF statement, etc so internally the representation of the data would be equally big as when it used as input a one-hot matrix because the model would need to know how it should react to every one of the categories. "No representation" at best means that the representation is learned by the algorithm, so skipping this step doesn't solve any problem, just makes life harder for your algorithm.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.