I already referred the posts here but this question is different. I don't wish to use categorical encoding. details given below
I have a dataset of 3000 unique customers purchase data. The dataset size is 380276.
I am working on a binary classification. The purchase data includes info on the below items
a) Name of the product (there are more than 1000 unique products)
b) segment to which the product belong to (there are 121 unique segments)
c) brokers who sold the product to customer (there are around 30 brokers)
d) Which country the customer belong to (there are more than 30 countries)
So, now I understand that one-hot encoding is not applicable (due to curse of dimensionality) and I don't wish to use hash-encoding (interpretability issue), label/ordinal encoding (imposes a rank order), target encoding (data leakage issue) etc.
So, I wish to transform high cardinal variable without using any encoding approach
In my dataset, a customer can have 300-400 transactions and they might have bought same product 400 times or 400 unique product (in 400 transactions).
Similarly, under a single product segment, there are multiple products. Of course, the level of cardinality of segment is less than products.
but what we think majorly drives the outcome is
product. So, we wish to know whether a customer buying a specific product makes him a likely to stay with us or not. I don't know how can I reduce the cardinality of this
For ex: We have info on frequently purchased product by the customer. For instance, I know that
customer A bought product A 60% of time and product B 20% of time and product C,D,E,F for 5% of the time. Not sure whether this qualifies as data leakage. Because, I am able to look back at the historical data now and compute the products that are bought most hy the customer.
Do you think knowing this, can help us reduce the cardinality of product variable for each customer? For ex: Can I create a column called
Top product purchased for each customer? and
Top 2 product purchased for each customer?
Can help me with the above?