1
$\begingroup$

I have 5400 rows of data and 3211 columns of attributes.

The first 4 columns are ID/Name/ParentID/ObjectType - the rest of the 3207 columns are the attributes that are to be used for similarity measures.

Huge dimensionality, I know, but I wanted to (as a first step) just see how this data clusters and finds similarity between all attributes.

I converted all attributes values to "0" if there was no value, and "1" if there was a value. I thought it'd be an easy first step to get me started with a clustering visual and similarity metric if I converted the values to binary.

Jaccard similarity seems to be a good measure for binary, but I'm stumped as to how to implement this (in Python) when I don't have any lists for comparison. Am I supposed to hard code each variable into the algorithm (3207 variables)?

I'm not sure where to start. Also, if there's a better way of doing this, I'm all ears. I've been researching how best to tackle this problem and there's so many similarity metrics that can be used, but I'm stuck on how to start since I there are so many columns that need to be used.

Added update: This is for retail product data. Each product has a mixture of boolean/categorical/numerical/meta data, and can have up to 3200+ attributes; however, not all of the attributes have a value.

The question needing research is whether or not similar attributes can define the category to which the product should live.

Jaccard Similarity to find similarities between sets. I view the columns in my dataset as the separate sets because I'm wanting to see if two products in the dataset are similar due to the attributes having the same binary value, or not (i.e.: productA has attributes 1,2,3,6 = 1; productB has attributes 1,2,3,5 = 1 - this could mean there's a 75% chance these two products are similar and could be put into the same category).

$\endgroup$
  • $\begingroup$ Can you describe the nature of the other attributes a little bit? $\endgroup$ – Zen May 15 at 18:46
  • $\begingroup$ Hi Zen, I've updated my original post with more info about the data/attributes. Let me know if this answers your question. $\endgroup$ – JessicaRabi May 15 at 19:45
1
$\begingroup$

Python is cool, but... Use R!

library(philentropy)
X <- matrix(rnorm(50), ncol = 5)
X[1, 2] <- NA
X[3, 5] <- NA
X[7, 5] <- NA
(X <- ifelse(is.na(X), 0, 1))
D <- distance(X, method = "jaccard")
(k_means <- kmeans(D, centers = 2, nstart = 100))
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.