# Clustering algorithms puts data points that are visually far apart in same cluster

I am trying to cluster a very large set of data points, of roughly (20000, 100) shape. I could not run density based DBSCAN or SpectralClustering due to the enormous amount of memory required. However when I do K-Means (with appropriate K selected with the 'elbow' method) or GMM, I see that points that are visually far apart and should not be labeled in a single cluster are grouped together.

My guess is such clusterings are not good for sparse data. Any suggestion would be great.

I'm attaching the clustered scatter plot (X, Y are normalized to fit in [0, 1]) to show what I am saying. I varied the random_sate, n_init but results are similar.

GMM

Edit:

Regarding X and Y, these are coordinates in 2 D plane. urrently I have about 20000 X values, and 100 Y values. How I represented the data in coordinate form is by taking all possible pairs out of these.

So in this representation it is 2 dimensional, not high dimensional. But number of coordinate is large.

• What is X and Y in the plot?
– Tim
Jun 20, 2023 at 6:05
• "the clustered scatter plot" how did you create this plot out of 100 columns? Jun 20, 2023 at 7:28
• How did you or the algorithm decide on the number of clusters? Jun 20, 2023 at 7:28
• For K-means by the popular elbow method. For GMM it is trial and error. I could not use density based approaches that takes epsilon distance as parameter. Those doesn't converge in reasonable time. Jun 20, 2023 at 10:27

• Yes, this representation is low dimensional. But you have 100 dimensions that you cluster over, and that is where the problems come from. Or not? What else precisely do you mean by "100 Y values"? Jun 20, 2023 at 10:42
• The 20000*100 (X, Y) pairs can be considered as location coordinates on ground. Jun 20, 2023 at 11:23
• say for k-means I did coords = numpy.vstack([X, Y[) # a (2000000,2) array Kmeans(coord, other params) Jun 20, 2023 at 11:26