Jaccard Similarity with Binary Data

Question

I have 5400 rows of data and 3211 columns of attributes.

The first 4 columns are ID/Name/ParentID/ObjectType - the rest of the 3207 columns are the attributes that are to be used for similarity measures.

Huge dimensionality, I know, but I wanted to (as a first step) just see how this data clusters and finds similarity between all attributes.

I converted all attributes values to "0" if there was no value, and "1" if there was a value. I thought it'd be an easy first step to get me started with a clustering visual and similarity metric if I converted the values to binary.

Jaccard similarity seems to be a good measure for binary, but I'm stumped as to how to implement this (in Python) when I don't have any lists for comparison. Am I supposed to hard code each variable into the algorithm (3207 variables)?

I'm not sure where to start. Also, if there's a better way of doing this, I'm all ears. I've been researching how best to tackle this problem and there's so many similarity metrics that can be used, but I'm stuck on how to start since I there are so many columns that need to be used.

Added update: This is for retail product data. Each product has a mixture of boolean/categorical/numerical/meta data, and can have up to 3200+ attributes; however, not all of the attributes have a value.

The question needing research is whether or not similar attributes can define the category to which the product should live.

Jaccard Similarity to find similarities between sets. I view the columns in my dataset as the separate sets because I'm wanting to see if two products in the dataset are similar due to the attributes having the same binary value, or not (i.e.: productA has attributes 1,2,3,6 = 1; productB has attributes 1,2,3,5 = 1 - this could mean there's a 75% chance these two products are similar and could be put into the same category).

Answer 1

Python is cool, but... Use R!

library(philentropy)
X <- matrix(rnorm(50), ncol = 5)
X[1, 2] <- NA
X[3, 5] <- NA
X[7, 5] <- NA
(X <- ifelse(is.na(X), 0, 1))
D <- distance(X, method = "jaccard")
(k_means <- kmeans(D, centers = 2, nstart = 100))