Cluster two feature samples with no knowledge of the number of clusters

Question

Thanks in advance for the help

I have around 13000 samples with two features each and I would like to cluster these samples into groups. A few caveats. One, I don't know how many groups there are (there could be as little as 70 to as many as 1000), but I do know a few things about the groups. The groups should be very, very similar across the two features. It is possible that several groups might be "close" to each other though. Two, there won't be any uniformity in terms of the number of elements in the groups. It is very possible (and from my knowledge of the dataset very likely) that some groups will have on the order of a hundred(s) of elements while others might have only a handful (possibly even one). It is not as important to identify the groups with a small number of elements, but erroneously adding them with a larger group could create noise for what I plan to do with these groups (this is not pertinent to this problem, I include this only as a constraint). Therefore I would like a method that will tend to err on the side of caution so to speak. Put simply, I would like to create a new group for any sample, or small set of samples, when it is not immediately clear that they belong to another group (when in doubt, just create a new group). I expect there to be many more groups with a low number of elements then groups with large elements. Hopefully this illuminates what I would like to accomplish accurately.

I believe the proper method to assigning these samples to different groups would be some sort of unsupervised clustering approach. However, I have very little experience with clustering. My questions are the following.

1. Is clustering the right approach here? If not, how else might I go about this problem?

2. Assuming that clustering is the right approach here, what method should I use? I understand that clustering algorithms often require the number of clusters in which to divide the samples. However, I don't even know the order of magnitude of the number of samples. Obviously, I could just create a 2d plot of the my samples and give a guesstimate, but I'm hoping for a method that would be a tad more rigorous and methodical then giving a guess. Are there any methods then that can dynamically create different numbers of groups given an input dataset and a set of tolerance settings?

Answer 1

For two-dimensional data, always use visualization.

Forget all the "internal evaluation" indexes to tell you how many clusters you have. Instead, look at the result to decide which result is interesting and insightful. The indexes just optimize some statistical number that may have 0 importance for your objective.

Yes, anything visual is the way to go, and your question would benefit from you adding an plot of the data.

Answer 2

Probably look at $k$-means clustering with two features, and calculate the (a) silhouette index, (b) Dunn's index, (c) Davies-Bouldin index, or (d) Hubert's $\Gamma$ for each value of $k$, which is preset before the clustering run. The peak value of silhouette as a function of $k$ might be the optimal number of clusters. The notion of finding the optimal number of clusters is called "cluster validity," and e.g. the silhouette index is one of the most commonly used metrics.