I've been asked to migrate this from StackOverflow to CrossValidated.

I have a problem that I think Machine Learning can solve but am having a very hard time determining which ML Algorithm to use and was hoping someone could lead me in the right direction. Here's a generalization of my problem using sports figures. I outsourced the reading/OCRing of PDFs to a company to scrape data off of image PDFs and send it to me in raw form so that I can then match that raw data to known entities.

I'm having the following 2 problems though, the text that is being scraped for the most part doesn't match up with what I was expecting and the OCR is giving me invalid characters. So lets say at the end of the day, I want to match a sports player to a team and all I am getting is the raw sports player name.

The data I am getting would look like this:

YearPlayed Name
1995       Mike Jordan
1995       M1chael Jordan
1995       1cha3l Jordan
2014       Tim Duncan
2014       Tomothy Duncan
2014       Peyton Manning
2014       Manning
2014       John Smith
2014       John A Smith

For each of those I have a list of all known teams and I would like to identify which team they play(ed) for. Ideally my data would've been a lot cleaner. I can do work normalizing it but the main problem I am having with that is the last example. For the sake of this argument, John Smith and John A Smith are two different people playing for two different teams and the only difference between them is a single character. So here are basically my questions:

  1. All of my data is textual data and I have a ton of classifiers, in the example above those players could play in any league with roughly 30 teams in each league, so lets say 5 leagues of 30 teams means there's roughly 150 classifiers, but in my actual data there are thousands of entities these could match to. Is there an ML algorithm that can take just text vectors as an input and match them to one of a very high number of classifiers?

  2. With enough training data, would any ML algorithm be smart enough to consistently differentiate between John Smith and John A Smith while knowing that Mike Jordan, Michael Jordan, M1chael Jordan and 1cha3l Jordan are all the same person?

  3. Ideally I would like to use R to do this in, if there is a viable ML algorithm that can handle this scenario, do you know if there is an R package that handles it?

  4. If ML isn't the right choice for this problem, what road should I be looking at going down?

One last bit of info on this, roughly 1% of my data has been matched and verified so I have a small amount of training data to start with. I also have handful of other features that I could add to the data so that it isnt just Name and Year but that data is textual as well. I can get the resources to increase this over time if I can prove that ML is the right course of action so I can eventually add more and more training data as time goes on.

  • $\begingroup$ Do you just have the names and years, or also have some textual context information? $\endgroup$ – xgdgsc Nov 28 '14 at 0:37
  • $\begingroup$ Entity resolution is a hard problem in large data sets. For example, I think there's also a Michael Jordan in the statistics/ML community, so how do you tell a computer that Michael Jordan == Michael Jordan sometimes, but sometimes Michael Jordan != Michael Jordan? You need a dedicated tool for ER. It's not really a ML task, just a computer program. $\endgroup$ – Sycorax says Reinstate Monica Sep 3 '15 at 14:12

For treating John Smith and John A Smith equal, you can take help from BMEWO encoding by considering 'A' as a middle name and thus keeping it optional. Then you can try normalizing your data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.