0
$\begingroup$

I'm working on a project where I have to develop a machine learning solution. I should mention that I'm not very experienced in ML yet and that I'm not even sure if the term "similarity function" is correct in this context.

I have profiles of applicants which include keywords for their skills. I have to use machine learning to cluster those keywords which often appear together into groups, as they indicate that those skills are closely linked to each other (example: people who have skills in Java often state C# skills as well, as the two languages are very similar).

The clustering should be no problem. In order to do this, my idea is to write a "similarity function" for these keywords. Of course this shouldn't be a string distance function, but a function that results in a high similarity for two keywords if they often appear together (and further modifies the results over time as the algorithm is fed with new profiles). And this is where I'm stuck as I'm not quite sure on how to approach this problem. I don't know how to do this for thousands of keywords in order to enable the use of a clustering algorithm.

I would be glad if you could give me some general advice or maybe suggest some existing solutions for this problem if there are any (I couldn't find anything related to this specific problem so far).

Thanks in advance!

$\endgroup$
0
$\begingroup$

Out all the data into an inverted list index.

Then you can efficiently compute the similarity of any two keywords as their relative intersection sizes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.