I am new here - and relatively new to statistics, data mining and R. I am trying to understand why my data is not clustering correctly - or if I am reading it wrong. Shortly about the project:

My data points are behavior of online gaming users over time (one observation for each user). I am trying to cluster a matrix containing 5000 observations. So far I have 4 dimensions that correspond to average games/day, interval between days, score, friends - all of these happened in the first 2 months of user's online presence. So it looks approximately like:

  avg_g_day avg_interval  score friends
1       8.5            6   6050       0
2   48.1304       1.8636  90530       0
3   70.0702       1.0714 293520       2
4        25            1   5710       4
5      3.75           10  10900       0

I am trying cluster this using all possible methods in R. So far I have not had success. I first scale my data using scale() and then fit the clusters. When I try kmeans (which would mean Euclidian distance, in my understanding), I get intersecting clusters upon visualization with clusplot. When I am trying to find the best k, I get that the best k is 2, which seems quite un-commonsense with 5000 points and 4 dimensions. I have tried other methods and other distance metrics, so far to no avail - I have used Mclust() and hclust. In each dimension the distribution is not uniform and I do not get warnings that best number of clusters is 1. Then why does the kmeans or pam method give me clusters that intersect on the 2 first principal components graph (clusplot)? Is there a way to separate them?

Since I am not sure which details could provide more insight into my problem, I will post them if questions arise. Thank you for any help.

  • $\begingroup$ Welcome to the site and thanks for providing context. I did some editing on your question; I hope I clarified it rather than making it into something you don't want. Why do you think 2 clusters is "un-commonsense" with 5000 points and 4 dimensions? One situation where this would be quite commonsense is if you had body measurements on 5000 adults, some men and some women. "Commonsense" applies more to the context than the number of dimensions or observations. $\endgroup$ – Peter Flom - Reinstate Monica Oct 2 '12 at 10:55
  • $\begingroup$ Hello Peter. Thank you for your comment - I am trying to say that my points are quite diverse. True is that most values are pretty close to 0 with some being remotedly far away (should I consider them outliers?). But even after I remove outliers and scale, the picture that clusplot gives me is k instersecting ellipses. I do not understand how to read such a plot. Perhaps they intersect only in the projection to 2 princomps? But I have read in some articles online that intersecting clusters on the clusplot mean that the clustering went unsuccessfully. $\endgroup$ – zima Oct 2 '12 at 11:15
  • 1
    $\begingroup$ There could well be only a single cluster in your data set. You can visualize 4 dimensions still pretty good. Do you visually see more than 2 clusters? $\endgroup$ – Anony-Mousse Oct 2 '12 at 12:22
  • $\begingroup$ Could you please give me a hint at how I could visualize 4 dimensions in R? I know only of 3D scatter plots, but 4 dimensions is a mystery for me. $\endgroup$ – zima Oct 2 '12 at 12:29
  • $\begingroup$ Instead of thinking about the points, think about what the clusters mean. Look at members of each cluster. Do they "make sense" together? As for 4-D, one way is lattice plots (see the lattice package). There are also ways to do this with different attributes (e.g. color), either in base graphics or ggplot. $\endgroup$ – Peter Flom - Reinstate Monica Oct 2 '12 at 12:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.