0
$\begingroup$

If you need to cluster a dataset with the following characteristics:

  • It has a mix of binary and continuous features.
  • It is very sparse. For most features, you only have values for 15% of the population. A value of 0 in one of the features, means that we don't know the actual value of that feature and not actually 0.

Here is what has been tried:

  • Running kmeans and agglomerative clustering after on both features after binarizing the continuous features. The results are two cluster with no clear seperation between the two. The choice of cluster is according to the best silouhette score. Kmodes has also been tried with similar outcomes.

  • Clustering the continuous features on their own using kmeans after log scaling, removing outliers. This yielded 5 clusters with clearly separable traits according to cluster means in each set of the continuous features.

  • Running kmodes on the binary features uniquely yielded 2 clusters that are clearly seperable and output a nice TSNE plot.

Given this:

  • What other algorithms/feature preprocessing techniques do you suggest that can solve a mixed dataset, given that making all variables categorical did not work so well and so far only treating them as two separate datasets worked?

  • How to handle missing values (in here represented as 0), since especially for binary features where a one only means that we are certain that the user has that trait and we cannot rule out the possibility that the other user who has 0 may actually be a one in that feature?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
2
$\begingroup$

One way to handle large number of binary features is to use topic modeling. In your case what I'm outlining can be applied just for the binary features, and then you can concatenate output with continuous features.

Topic modeling is indeed used for reducing dimensionality of sparse data, document-term matrices for text. It is soft clustering (because documents are transformed into distributions of topics).

For example see this vignette from scikit-learn documentation

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.