I need some help finding a better way to statistically group data for a project I am working on.

I have a data set of individuals with different skill sets. Each individual can have only 1 or several different skills. Some of the individuals can even have up to 40 skills that they work with. I am trying to find a more statistical way to find groups of agents based on the skill while also keeping a large percent of the data. Here is what the data looks like:

       Individual ID    Skills 
            1              1,2,3,4     
            2              2,3,4
            3              2,3
            4              3,4

So ideally, it would be nice to find the maximum number of skills that group people together and maintain the most number of records. What would not be ideal is taking all of the skills because some of the skills may have very low pct of the individuals or records. Is there a more statistical approach to grouping these people together into one large bucket (for example, we use 2,3,4 because its 90% of the agents and 95% total records)

Another good question that I have still yet to figure out is what percentage of coverage is good.


  • 1
    $\begingroup$ Why are you trying to do this? Presumably there is some analysis, inferential modeling, prediction modeling, or data presentation (or you want to more generally answer some particular question), right? E.g. is it about finding groups of people and describing them in terms of their skills? Or perhaps more about the skills and which ones often go together? Depending on what exactly you want to do, it may be that different approaches could be desirable. E.g. there's probably a lot of sensible ways one could create embeddings spaces and then do clustering. $\endgroup$
    – Björn
    Jan 12, 2023 at 9:03

2 Answers 2


Sounds as if you are looking for a relaxed version of the

set cover problem

which is np hard. But depending on how you relax it, an effective greedy approach might work.

More information: https://en.m.wikipedia.org/wiki/Set_cover_problem


Your dataset lends itself very naturally to the Correlation Explanation (CorEx) approach that I've been using for a while. The idea is to build representations that are maximally informative about data using entropy-related measures. The beauty is that no strong assumptions are made about the data.

Please check out the papers and source code here. You might be also interested to read more publications about the subject by the same author here.

For your categorical data, which is exactly the data that the framework accepts, you can get the groupings for users and/or skills. Exactly what are you looking for.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.