10
$\begingroup$

I have a dataset of 130k internet users characterized by 4 variables describing users' number of sessions, locations visited, avg data download and session time aggregated from four months of activity.

Dataset is very heavy-tailed. For example third of users logged only once during four months, whereas six users had more than 1000 sessions.

I wanted to come up with a simple classification of users, preferably with indication of the most appropriate number of clusters.

Is there anything you could recommend as a solution?

Improve this question
$\endgroup$
2
  • $\begingroup$ I don't see why you want to cluster your users without any subjective input (I refer to a comment you made @reed), you said you want "the most appropriate number of clusters", but unfortunatly you don't have a clear objective, if you want to cluster your population to show something particular, you should tell us what you want to show ? If you want statistic (the data) to tell you what you want to show this is another problem :) $\endgroup$
    – robin girard
    Commented Jul 25, 2010 at 11:24
  • $\begingroup$ @robin: thanks for comment. i haven't seen much research in this area and the input from data provider was minimal [so far they were not interested/capable of investigating it further]. after initial exploration of data it was quite clear for me that there are couple of distinctive patterns [examples being 'heavy downloaders' in just a few locations or 'frequent hoppers' with lots of small sessions in large number of locations]. my goal at this stage was to try to use data itself to tell me how can i divide it best and minimize subjective input. $\endgroup$
    – user22
    Commented Jul 27, 2010 at 14:59

4 Answers 4

Reset to default
3
$\begingroup$

You could of for a supervised self-organizing map (e.g. with kohonen package for R), and use the login frequency as dependent variable. That way, the clustering will focus on separating the frequent visitors from the rare visitors. By plotting the number of users on each map unit, you may get an idea in clusters present in your data.

Because SOMs are non-linear mapping methods, this approach is particularly interesting for tailed data.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks Egon. SOM approach might be very interesting indeed. Will have a look at it. $\endgroup$
    – user22
    Commented Jul 20, 2010 at 16:23
2
$\begingroup$

K-Means clustering should work well for this type of problem. However, it does require that you specify the number of clusters in advance.

Given the nature of this data, however, you may be able to work with a hierarchical clustering algorithm instead. Since all 4 variables are most likely fairly highly correlated, you can most likely break out clusters, and stop when you reach a small enough distance between clusters. This may be a much simpler approach in this specific case, and allows you to determine "how many clusters" by just stopping as soon as you've broken your set into fine enough clusters.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks Reed. k-means is definitely a way to go and I explored several solutions. However I am trying to go for a solution without subjective decision of the number of clusters. Hierarchical approach on the other hand produced huge classification and again it was hard for me to specify where to stop. $\endgroup$
    – user22
    Commented Jul 20, 2010 at 16:22
2
$\begingroup$

You might consider transforming (perhaps a log) the positively skewed variables.

If after exploring various clustering algorithms you find that the four variables simply reflect varying intensity levels of usage, you might think about a theoretically based classification. Presumably this classification is going to be used for a purpose and that purpose could drive meaningful cut points on one or more of the variables.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks Jeromy. I explored several possible transformations of data using Stata's gladder function for the ladder of powers. Teoreically based solution will be one way to go, but before that I wanted to check for the data driven solution. $\endgroup$
    – user22
    Commented Jul 20, 2010 at 16:19
2
$\begingroup$

This is problem with clustering, you can't tell what is considered a cluster. I would see what is the reason behind clustering users, specify my threshold value, and use hierarchical clustering.

In my experience, one has to set either the number of cluster needed, or the threshold value (the distance value that binds two data point together).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.