I have a dataframe of the form

| user_ID | cat1 | cat2 | ..... |cat 2125 |
   234       1.0    0.0            0.0
   125       0.0    1.0            0.0
   542       0.0    1.0            1.0

This dataset represents user votes on 2125 books. Each category column stands for one book. It contains either a 1.0, for a vote given by that user or a 0.0, for no vote.

I want to cluster these users by the books they voted for. I realize I first need to reduce dimensionality. I tried PCA and TruncatedSVD, with k-Means clustering afterwards. In both cases, the clusters I get are irrelevant (users usually have no books in common).

My only constraint here is that i must use k-Means (since I need the coordinates of the centroids). I suspect that my dimensionality reduction is not good though.

Can someone please offer some advice as to what I should do?


I am no expert, but in this situation, I'd might try other dimensionality reduction techniques if those previous ones did not work.

There are methods such as KernelPCA, using different kernels, with which you can even construct your own. There are other dimensionality reduction techniques - this link https://towardsdatascience.com/dimensionality-reduction-for-machine-learning-80a46c2ebb7e might give you a start.

Did you plot them up in 2-3 dimensions? If you plot up several different techniques, you can see which dimensionality reduction technique might be the most suitable for obtaining your clusters - this assumes that your dataset is not too big. You could possibly go up to 4 dimensions by using a colour scheme for the 4th dimension. Although 2-4 dimensions might not capture enough variance, you often do not want to go into a high space of dimensions with k-means as it suffers from Curse of Dimensionality. Check how much variance you capture and pick a suitable number, but I would definitely not try and cluster in 2125 dimensions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.