Does clustering need scalar data?

Question

I am trying to cluster 43,000 individuals on about 50 variables. The data contained in the variables are minutes of a radio shows which people listened to in the range of 0 - 3,000,000 minutes. My objective is to arrive at clusters of individuals based on the number of minutes of shows viewed and the group of shows viewed by them. Data looks like below:

    Show1   Show2   Show3 
ID1 382900  24597   1456
ID2 12678   23246   75436
ID3 85304   5754    966485
ID4 750923  5656    2478
ID5 483     23578   45245
ID6 37505   0       0

Unfortunately, when I try the kmeans or Ward methods using PROC FASTCLUS and PROC CLUSTER in SAS, the bulk of the individuals (90%) get classified into a single group. When I observe the dataset, I feel that there are significant differences in magnitude of minutes across individuals hence it's not possible that they all belong to the same cluster.

I've tried many ways:

1. clustering the raw data,
2. doing factor analysis, calculating factor scores and clustering the factor scores instead of the raw data.
3. Filtered the dataset - cleaned sparsely populated variables,
4. varied the number of clusters based on many criteria from 3-4 clusters to as many as 20-30 clusters
5. clustered on unstandardized and standardized data, etc.

All have failed to give me disjoint clusters but I am not convinced that 90% of my population behaves similarly.

What should I do? I thought of assigning values in the range of 1-7 based on the range under which each value falls for e.g. 0-1000 minutes = 1, 1000 - 50000 minutes = 2 etc. But then why do I need the clustering?

Answer 1

The results of clustering depend significantly upon the similarity metric (which points are "close," and which "far"), and the metric may depend upon the units used for each variable, weightings, and so on. If the average value of one feature value (your Show 1 minutes, for example) is much larger than others, clustering will depend upon that. Depending upon your application, you might want to treat each show's effect "equally," in which case you should rescale each column or whiten your data (make each have zero mean and unit variance). In sum, data representation and preprocessing (e.g., scaling) will depend upon your task at hand.

Answer 2

Both k-means and ward minimize squared deviations.

In other words, they assume that the deviation from 0 to 10 minutes is the same as from 1000 to 1010 minutes, and the difference 0 to 20 minutes is 4 times as much.

This may not hold for your data. Consider spending a lot of time to evaluate similarity, and then use complete linkage clustering with the fine-tuned distance.