1
$\begingroup$

I am trying to cluster 43,000 individuals on about 50 variables. The data contained in the variables are minutes of a radio shows which people listened to in the range of 0 - 3,000,000 minutes. My objective is to arrive at clusters of individuals based on the number of minutes of shows viewed and the group of shows viewed by them. Data looks like below:

    Show1   Show2   Show3 
ID1 382900  24597   1456
ID2 12678   23246   75436
ID3 85304   5754    966485
ID4 750923  5656    2478
ID5 483     23578   45245
ID6 37505   0       0

Unfortunately, when I try the kmeans or Ward methods using PROC FASTCLUS and PROC CLUSTER in SAS, the bulk of the individuals (90%) get classified into a single group. When I observe the dataset, I feel that there are significant differences in magnitude of minutes across individuals hence it's not possible that they all belong to the same cluster.

I've tried many ways:

  1. clustering the raw data,
  2. doing factor analysis, calculating factor scores and clustering the factor scores instead of the raw data.
  3. Filtered the dataset - cleaned sparsely populated variables,
  4. varied the number of clusters based on many criteria from 3-4 clusters to as many as 20-30 clusters
  5. clustered on unstandardized and standardized data, etc.

All have failed to give me disjoint clusters but I am not convinced that 90% of my population behaves similarly.

What should I do? I thought of assigning values in the range of 1-7 based on the range under which each value falls for e.g. 0-1000 minutes = 1, 1000 - 50000 minutes = 2 etc. But then why do I need the clustering?

$\endgroup$
  • 2
    $\begingroup$ Does your similarity work? If your similarity does not work, the clustering algorithms don't have a chance. First solve how to identify similar users then cluster. $\endgroup$ – Anony-Mousse Jan 23 '15 at 21:51
1
$\begingroup$

The results of clustering depend significantly upon the similarity metric (which points are "close," and which "far"), and the metric may depend upon the units used for each variable, weightings, and so on. If the average value of one feature value (your Show 1 minutes, for example) is much larger than others, clustering will depend upon that. Depending upon your application, you might want to treat each show's effect "equally," in which case you should rescale each column or whiten your data (make each have zero mean and unit variance). In sum, data representation and preprocessing (e.g., scaling) will depend upon your task at hand.

$\endgroup$
  • $\begingroup$ ? But i've tried it with scaling already ! It hasn't worked. My question is what I can do to improve the clustering given that nothing I've tried has worked so far . . . $\endgroup$ – vagabond Jan 23 '15 at 19:32
  • $\begingroup$ Please define "improve" specifically and quantitatively... without that, nobody can help you. You want merely "disjoint" clusters? That is guaranteed by kMeans and numerous other techniques. If you can point us to data, we can try to cluster it. $\endgroup$ – David G. Stork Jan 23 '15 at 21:16
  • $\begingroup$ Re: your 2nd sentence: are you sure? I have not found this to be true. $\endgroup$ – rolando2 Jan 24 '15 at 4:18
  • $\begingroup$ @rolando2 I agree. It is not true. absolute difference does not play much role. $\endgroup$ – vagabond Jan 25 '15 at 2:47
  • $\begingroup$ @vagabond Any improvement seen in doing clustering. I too facing similar problem. Can you let me know what you did to cluster $\endgroup$ – StatsUser Jun 16 '17 at 2:22
1
$\begingroup$

Both k-means and ward minimize squared deviations.

In other words, they assume that the deviation from 0 to 10 minutes is the same as from 1000 to 1010 minutes, and the difference 0 to 20 minutes is 4 times as much.

This may not hold for your data. Consider spending a lot of time to evaluate similarity, and then use complete linkage clustering with the fine-tuned distance.

$\endgroup$
  • $\begingroup$ thanks for the tip. will try complete linkage too though given the size of my dataset - wonder if it will take too long to complete. $\endgroup$ – vagabond Jan 25 '15 at 2:47
  • $\begingroup$ Start with a sample. If it doesn't work there, it won't work on the full data set either. $\endgroup$ – Anony-Mousse Jan 25 '15 at 9:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.