Converting unsupervised to supervised problem - Overfitting - bad?

Question

I am working on a customer segmentation using 5 features such as recency, frequency, monetary, tenure, unique_product_cnt etc.

So, I did a RFM based segmentation where I used jenks optimization to find clusters/groups of each of the variables.

Later, based on some if-else condition, I created labels (segment names) for these customers. I have a dataset of 2500 records where 1100 belongs to label loyal and 1400 belong to label At Risk.

Now, I am trying to find out what are the features impact/contribution of features to the output label (cluster). like ordering features based on their importance.

So, using the label that I derived, I built a supervised learning model using random forests.

I am building random forests only to understand the feature importance and find out which features were more important in driving the outcome/class label.

So, I did a train, test split and gridsearch over best parameters.

Unfortunately, I see that I get 100% accuracy, f1, recall etc in both train and test

While I don't intend to build a supervised model but my requirement is just to understand what features/characteristics contributes a customer to be either loyal or At Risk. Basically, to understand my cluster formation, I built a supervised model.

So, for this purpose is it okay to live with 100% performance across all metrics?

Or is it cause of concern? Am I making any mistake?

You can find my f1-score threshold optimization below

Answer 1

Because each cluster will be a high-density region, and this region is "pure" with respect to its cluster label, and because the clusters are distinguished by some partitioning of the feature space, it's trivial to train a classifier to recreate the clusters' boundary. It's not surprising that the random forest was able to capture the boundary exactly, because we know by construction that the labels determined by clustering correspond to a partition of the data according to the feature space.

Typical reasons that a classifier is not perfectly able to predict the label is due to some combination of

• randomness in the relationship between features and labels, and
• the features are not highly informative.

But we know that in this application, the features must be informative (you're using the features to make your cluster assignment labels and the same features to classify) and the labels cannot be random (any two observations in the same cluster must have the same label, by construction).