I have a dataset with 120k rows each one representing a job and I can have as many as 200 columns (these are the skills required to complete the job). S_ij=1 if the jth skill is required to complete job J_i, 0 otherwise. And I also have the category for that job (total of 24 categories), let's call it C_i. I want to form clusters of categories based on the skills required by each job.
What approach would you suggest to conduct this? I was trying to use some Hierarchical clustering with a distance metric suitable for mi case.
Since my data set is quite large, I guess the dendogram is going to be unreadable and useless.
What is the best approach to do hierarchical clustering when the data set is quite large?
How would you approach this problem?