I am trying to cluster 43,000 individuals on about 50 variables. The data contained in the variables are minutes of a radio shows which people listened to in the range of 0 - 3,000,000 minutes. My objective is to arrive at clusters of individuals based on the number of minutes of shows viewed and the group of shows viewed by them. Data looks like below:
Show1 Show2 Show3 ID1 382900 24597 1456 ID2 12678 23246 75436 ID3 85304 5754 966485 ID4 750923 5656 2478 ID5 483 23578 45245 ID6 37505 0 0
Unfortunately, when I try the kmeans or Ward methods using
PROC FASTCLUS and
PROC CLUSTER in SAS, the bulk of the individuals (90%) get classified into a single group. When I observe the dataset, I feel that there are significant differences in magnitude of minutes across individuals hence it's not possible that they all belong to the same cluster.
I've tried many ways:
- clustering the raw data,
- doing factor analysis, calculating factor scores and clustering the factor scores instead of the raw data.
- Filtered the dataset - cleaned sparsely populated variables,
- varied the number of clusters based on many criteria from 3-4 clusters to as many as 20-30 clusters
- clustered on unstandardized and standardized data, etc.
All have failed to give me disjoint clusters but I am not convinced that 90% of my population behaves similarly.
What should I do? I thought of assigning values in the range of 1-7 based on the range under which each value falls for e.g. 0-1000 minutes = 1, 1000 - 50000 minutes = 2 etc. But then why do I need the clustering?