0
$\begingroup$

I have 2,000 people with 60 features, the feature are highly correlated. I can't remove the correlated features in my case: I need all the data. I want to group similar people using a clustering algorithm.

I have tried:

  1. k-means using Euclidean distance on "max-min" normalized data: the distribution between the group was somewhat even.

  2. h-clust using mahalanobis distance on Moore-Penrose generalized Inverse matrix of the data: the distribution between the group was not good (more then half was in one group and all the other had very little people in it (1 or 2 people in some).

From what I have read for correlated data, it is best to compute distance using mahalanobis distance, but the result was very odd.

Did I compute it right?

Do you have other suggestion to cluster the people when the data is correlated?

Improve this question
$\endgroup$
10
  • $\begingroup$ How many PC's does it take to account for 95% of the variation? $\endgroup$
    – generic_user
    Commented Feb 14, 2017 at 15:18
  • $\begingroup$ @generic_user how can I know that? it's in the function "kmeans" and "hclust"? $\endgroup$
    – HilaD
    Commented Feb 14, 2017 at 15:43
  • $\begingroup$ There is a close link between mahalanobis distance and principal components analysis. I'd suggest that you read up on it. Basically you determine the orthogonal directions in your dataset that successively explain the most variability. In R, the function prcomp implements it. But without reading up, you'll have no idea how to interpret the output. $\endgroup$
    – generic_user
    Commented Feb 14, 2017 at 17:43
  • 2
    $\begingroup$ What do you mean PCA "didn't work"? If your first few PC's account for almost all of the variation, then you might consider clustering only on those (derived) varibles. $\endgroup$
    – generic_user
    Commented Feb 14, 2017 at 19:28
  • 1
    $\begingroup$ I don't know without knowing more about your context. Sometimes clustering is very difficult, and doesn't work cleanly. $\endgroup$
    – generic_user
    Commented Feb 14, 2017 at 20:39

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.