I want to create an ML program that cleans up noisy data. I have the raw text features and the labels. ML programs tend to prefer numeric data, so I need to encode my text features. This is an example of how my data looks like:

**Raw text feature**                            **Label**
10/01/2017-XYZ123456-JOHN DOE - ABC2345         Doe, John
01 02 14 - J A N E D O E _(temp) QWERTY12       Doe, Jane
01feb2018#02mar2018#john#smith#ID12345          Smith, John

My first step is to find the right encoder. I do not want a one-hot encoder because there is a large, finite and undefined set categories/Labels. I need a more dynamic encoding that can vary in size and works great with data it has not seen before. Question 1: Any suggestions on how to encode this data?

Then I have to choose an ML classification method. I would go with neural networks because of the complexity of the feature. The text feature is a sequence of characters which I am hoping my ML program would be able to predict (or extract) a name in the form [Last Name, First Name]. I would use a Recurrent Neural Network, for which I will figure out the optimal hyperparameters once I encode the text data properly.

Question 2: Any suggestions on an alternative to a recurrent neural network? Maybe an approach outside neural networks?

Thanks for your help!

  • 1
    $\begingroup$ What does " clean up noisy data" mean? $\endgroup$ Jul 20, 2018 at 3:11
  • $\begingroup$ In this case, the noise in my data is all the unwanted characters in my string. The data is clean once all of this is removed and the only part of the string that remains is the person's name. $\endgroup$
    – Agarp
    Jul 20, 2018 at 13:40

1 Answer 1

  1. You can use BIO encodding (https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)): mark whole the region that contains names with Is (including separators - I would deal with them later) and the rest characters with Os plus B for the Beginning.
  2. This is a problem of sequence labelling there is a number of alternatives to RNN, like LSTM, HMM and variations, CRF and hybrids.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.