# K-means classifies 96% of my data in 1 cluster. Any suggestions to improve the results?

Problem: K-Means clustering shows 96% of my data belongs to one cluster. How can I improve my results or should I conclude that no cluster exists in my dataset. Dbscan clustering shows 1 cluster exists.

Dataset: I have monthly purchase data for 15 products across all customers for the past 15 months. Most of my customers would use a combination of these 15 products. However the proportion of different products purchased varies across customers and occasionally may vary over months. Usually the type of products a customer purchases is similar month over month. For example I have few customers who buy mostly P4 and P5 and very less of others, few customers only buy P10 and less of other products in comparison. Most of my customers will but 10-12 products. My customers usually make daily transactions and the data I have is aggregated monthly. There is huge variance across purchase data. So the (min, max) for P10 is (0,2.685250e+07 ) while (min, max) for P3 is (0, 5843). This min, max is across all customers, all months.

Methodology: I did a min-max scaling to normalize my dataset. However when I plot the scatter-plot, I don't see a normal curve (is it because this is count data and not continuous data?). After normalization, I did PCA analysis which reduced my feature set from 15 to 12.

I plotted the elbow curve. Below are two plots. Elbow curve plotted on scaled data:

Elbow curve on PCA reduced data:

I then did a K-means clustering on PCA reduced data and form two clusters. My cluster 0 has 3397 entries while cluster 1 has only 128 entries! Below is a scatter plot of my clusters:

How can I make better clusters? What algorithm should I use? I want to understand what product combinations are different customers buying. This doesn't identify my customers who mostly buy P10 and not much of others and similar customers.

• This seems impossible to answer because it's not obvious that there should be more clusters. Your plots certainly don't show much by way of structure. Such structure might exist even if we can't see it, which means you cannot really conclude that your result is the best possible either. One thing to consider trying is transforming one or more of your variables, because your final plot seems to indicate quite skewed distributions.
– mkt
Aug 28, 2019 at 4:18
• Your data cloud shape is very different from what k-means assumes as a cluster analysis technique. This is just to start with. Aug 28, 2019 at 4:38
• Thanks @mkt for your thoughts. I will try transforming some variables. As you rightly said, my distributions are very skewed (as seen in the scatter plots). I intuitively, from business expect that there are structures though I can't see it. Aug 28, 2019 at 5:18
• Thanks @ttnphns - any thoughts on what algorithm will be best suited for this? Aug 28, 2019 at 5:18
• Try DBSCAN or nearest neighbour hierarchical clustering. Aug 28, 2019 at 5:40