Correct clustering approach for segmenting stores

Question

Domain : Retail I have a set of stores which I want to cluster into similar stores based on 10 variables: revenue, avg income, market share etc. I took 2 approach:

Approach 1: Given there are 10 dimensions its a bit hard to visualize them so I standardised them using standardscaler() and then used tsne to visualize them in 2D. This is how it looks:

The data distribution looks spherical and so I applied kmeans with metric as euclidean on this transformed data.

Approach 2: I did not apply any high dimensional transformation such as pca or tnse and instead just standardised the data and ran clustering algorithms such as HDBSCAN, kmeans, hierarchical etc. The thing is with DBSCAN I am getting all the stores as outliers i.e. cluster -1. With algorithms such as kmeans etc the silhouette cofficient value is pretty less , like close to .15 etc.

I am confused as to how to go about clustering this data set and validating in a proper manner.

Data summary : 2000 data points, 10 variables which are all continuous, although a mixture of absolute number & percentages.

Answer 1

1. Spherical is not good for kmeans. That is, unless you see k spheres that have a similar diameter and that are well separated. Usually, clustering on top of a tSNE visualization is a bad idea, because the visualization does not preserve data density (on the contrary, for visualization purposes it tries to spread out the data!) - by using it, you often remove important signal from your data. Because tSNE reduces the data to 2d, you may just as well select groups in that plot by hand. There is some small cluster in the bottom right, maybe one in the top left. In the bottom left these are likely undesired artifacts. I don't see much more, so don't expect an algorithm to find better clusters on this 2d plot!

2. DBSCAN requires that you choose the parameters. That begins with choosing a distance function, but the most important parameter is the distance threshold epsilon. For some stupid reason, sklearn had a default value; unfortunately one that never works well except on 2d toy data. If the radius is badly chosen, you tend to get all points labeled as noise (-1) or all in one big cluster (all 0).

3. Have you considered that your data may not have larger clustering structures? Your application sounds more like there is no significant pattern to be expected, but that you just want to artificially break them apart?

4. Data preprocessing is important. Percentage variables should not be normalized. They already are. But other variables need to, monetary variables for example are often skewed and need to be treated carefully. Just to give examples. In the end, you should not be looking at some random measure such as Silhouette (which depends on your preprocessing, too, and may look better if you do bad preprocessing), but does all that make sense for your business? Write down the equation what you are measuring in the clustering algorithms - does it have relevance for your business?