I have a dataset with 120k rows each one representing a job and I can have as many as 200 columns (these are the skills required to complete the job). S_ij=1 if the jth skill is required to complete job J_i, 0 otherwise. And I also have the category for that job (total of 24 categories), let's call it C_i. I want to form clusters of categories based on the skills required by each job.

What approach would you suggest to conduct this? I was trying to use some Hierarchical clustering with a distance metric suitable for mi case.

Since my data set is quite large, I guess the dendogram is going to be unreadable and useless.

What is the best approach to do hierarchical clustering when the data set is quite large?

How would you approach this problem?


1 Answer 1


If you want to cluster the categories, you only have 24 records (so you don't have "large dataset" task to cluster). Dendrograms work great on such data, and so does hierarchical clustering.

I'd suggest to:

  1. flatten the data set into categories, e.g. taking the average of each column: that is, for each category and each skill divide number of 1's in the skill / number of jobs in the category.
  2. try different similarity measures, including Manhattan and Jensen–Shannon divergence.
  3. use hierarchical clustering, and visualize the dendrogram.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.