This has been inside my mind for at least a few hours. I was trying to find an optimal k for the output from the k-means algorithm (with a cosine similarity metric) so I ended up plotting the distortion as a function of the number of clusters. My dataset is a collection of 800 documents in a 600-dimensional space.
From what I understand, finding the knee point or the elbow point on this curve should tell me at least approximately the number of clusters I need to put my data into. I put the graph below. The point at which the red vertical line was drawn was obtained by using the maximum second derivative test. After doing all this, I was stuck at something much simpler: what does this graph tell me about the dataset?
Does it tell me that it is not worth clustering and that my documents lack structure or that I need to set a very high k? One strange thing though is that even with low k, I am seeing similar documents being clustered together so I am not sure why I am getting this curve. Any thoughts?