I'm working on a project where I have to develop a machine learning solution. I should mention that I'm not very experienced in ML yet and that I'm not even sure if the term "similarity function" is correct in this context.
I have profiles of applicants which include keywords for their skills. I have to use machine learning to cluster those keywords which often appear together into groups, as they indicate that those skills are closely linked to each other (example: people who have skills in Java often state C# skills as well, as the two languages are very similar).
The clustering should be no problem. In order to do this, my idea is to write a "similarity function" for these keywords. Of course this shouldn't be a string distance function, but a function that results in a high similarity for two keywords if they often appear together (and further modifies the results over time as the algorithm is fed with new profiles). And this is where I'm stuck as I'm not quite sure on how to approach this problem. I don't know how to do this for thousands of keywords in order to enable the use of a clustering algorithm.
I would be glad if you could give me some general advice or maybe suggest some existing solutions for this problem if there are any (I couldn't find anything related to this specific problem so far).
Thanks in advance!
