0
$\begingroup$

I need some help finding a better way to statistically group data for a project I am working on.

I have a data set of individuals with different skill sets. Each individual can have only 1 or several different skills. Some of the individuals can even have up to 40 skills that they work with. I am trying to find a more statistical way to find groups of agents based on the skill while also keeping a large percent of the data. Here is what the data looks like:

       Individual ID    Skills 
            1              1,2,3,4     
            2              2,3,4
            3              2,3
            4              3,4

So ideally, it would be nice to find the maximum number of skills that group people together and maintain the most number of records. What would not be ideal is taking all of the skills because some of the skills may have very low pct of the individuals or records. Is there a more statistical approach to grouping these people together into one large bucket (for example, we use 2,3,4 because its 90% of the agents and 95% total records)

Another good question that I have still yet to figure out is what percentage of coverage is good.

Thanks!

$\endgroup$
0
$\begingroup$

Sounds as if you are looking for a relaxed version of the

set cover problem

which is np hard. But depending on how you relax it, an effective greedy approach might work.

More information: https://en.m.wikipedia.org/wiki/Set_cover_problem

$\endgroup$
0
$\begingroup$

Your dataset lends itself very naturally to the Correlation Explanation (CorEx) approach that I've been using for a while. The idea is to build representations that are maximally informative about data using entropy-related measures. The beauty is that no strong assumptions are made about the data.

Please check out the papers and source code here. You might be also interested to read more publications about the subject by the same author here.

For your categorical data, which is exactly the data that the framework accepts, you can get the groupings for users and/or skills. Exactly what are you looking for.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.