1
$\begingroup$

This is a variation on my question Machine learning to catch fraud

I have a data set with about 5 million rows The data set contains names and addresses of companies The name of the company is free text as is are the addresses It seems some of the companies have very bad data. It has been suggested by my manager that a script is generating the information and is varying it slightly each time a company uses our services

The records look something like below Customer Name: ASDFGH KL: Shipper Contact Name ASDFLKJ:

Again its not neccessary that Customer and Shipper Name are roughly the same

I was wondering is there a machine learning algorthim that could flag these customers that anyone is aware of that looks like the users are just inputting random letters like above (To be fair, in this example, the letters are not random at all but all taken from the third row of the keyboard)

I was thinking of adapting spam filtering to try and identify the customers by first tokenizing the Customer Name and generating N-Grams and then classifying it as SPAM or NON SPAM using something like bayes. For this i would need a very large corpus which would neccesitate finding them in the original data set which could be very time consuming. I currently have a starting point of ten records where this is the case

Does anyone know if there is any method or theory to catch people submitting random letters?

$\endgroup$
1
$\begingroup$

Depends a bit on how globally applicable solution you want. One simple idea is to try to cross-check versus some other sources of information (perhaps allowing for a little bit of misspelling). If the customers are primarily from your country, what about comparing names to names in the telephone book (or some other sensible source of names that you can get electronically)? Only very few names will not occur in the telephone book. Similarly, companies are usually in the telephone book (or have an internet page). I guess there are also typical letter patterns with typical American/European names (e.g. you will rarely have more than 4 consonants before getting to a vowel etc.), but names from certain cultures may not follow the same patterns. In many countries streets are named after things or people that are likely to have wikipedia entries and/or to be in telephone book.

$\endgroup$
  • 1
    $\begingroup$ Hi Bjorn, I like the idea of the consonants very much, I could also look at if they have numbers in the field. All this information is coming from Asia so the variation on names spelt and mis-spelt will be huge. I will see if i can use the Google API to check for company names also. Thank you for your help. You have given me a lot to think about $\endgroup$ – John Smith Aug 31 '15 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.