If you need to cluster a dataset with the following characteristics:
- It has a mix of binary and continuous features.
- It is very sparse. For most features, you only have values for 15% of the population. A value of 0 in one of the features, means that we don't know the actual value of that feature and not actually 0.
Here is what has been tried:
Running kmeans and agglomerative clustering after on both features after binarizing the continuous features. The results are two cluster with no clear seperation between the two. The choice of cluster is according to the best silouhette score. Kmodes has also been tried with similar outcomes.
Clustering the continuous features on their own using kmeans after log scaling, removing outliers. This yielded 5 clusters with clearly separable traits according to cluster means in each set of the continuous features.
Running kmodes on the binary features uniquely yielded 2 clusters that are clearly seperable and output a nice TSNE plot.
What other algorithms/feature preprocessing techniques do you suggest that can solve a mixed dataset, given that making all variables categorical did not work so well and so far only treating them as two separate datasets worked?
How to handle missing values (in here represented as 0), since especially for binary features where a one only means that we are certain that the user has that trait and we cannot rule out the possibility that the other user who has 0 may actually be a one in that feature?