Suggested to ask here from https://stackoverflow.com/questions/56038093/short-string-classification-high-acc-tons-of-false-positives-are-we-on-the-ri (same question but on stackoverflow)

I've been working on feature extraction of documents with multiple frameworks for a few months, and recently the project has found a dead end.

I'm aiming to find any kind of identification string within the document.

Thanks in advance

Let's say that the project is structured into "modules" that are called separately, and the latest one that has been in development is aiming to find identification numbers within the document as I've said.

For example:

  • "A/364" is a valid identifier.
  • "137 . 23" is a valid identifier.
  • "05.08.2019" is not (It's a date).

To avoid applying the module to the entire document and enhancing the accuracy, I'm finding a label and extracting the text located near it either to its right or down below it (I'm following occidental reading order for the time being, left to right, top to bottom). THIS PART WORKS

Just for the sake of further explanation, imagine that we're going to extract dates, we'll find a Label corresponding to "date" or something along those lines and then we'd apply either regex or some other solution to find a date.

The thing is that there's not a normalized way of assigning an identifier to something, so you can expect ANY kind of string.

After that dead long introduction:

What I've tried: * Applying one or multiple regexes to the extracted text.

And the solution I'm currently trying (without results as the title says) * Convert the extracted text to a single string replacing each type of character with a generic and then apply n-grams with scikit.

With the following function, any given string is replaced so letters are "a", capital letters "b", digits "c", whitespaces "d" and so on...

def st_to_chars(in_string):
    in_string=re.sub("[a-z]", "a",in_string)
    in_string=re.sub(" ","d",in_string)

    return in_string

At first, it seemed like a good approach since we got an accuracy of 0.931280

#Create classifier and vectorizer
clf = MultinomialNB(alpha=0.1)
clf2 = LinearSVC(random_state=0, tol=1e-5)
vec2 = CountVectorizer(analyzer='char_wb', ngram_range=(2,4), min_df=1)

df = pd.read_csv("dataset.csv", delimiter = ",", quotechar='"')
df = df[pd.notnull(df['code'])]
df = df.sample(frac=1).reset_index(drop=True)

y_train = df['label'][0:45000].tolist()
data_train  =df['new_code'][0:45000].tolist()
x_train = vec2.fit_transform(data_train)
clf2 = LinearSVC(random_state=0, tol=1e-5)

y_true = df['label'][45000:].tolist()
data_test  =df['new_code'][45000:].tolist()
x_test = vec2.transform(data_test)

y_test = clf.predict(x_test)
sklearn.metrics.accuracy_score(y_true, y_test)

The dataset contains over 75k rows of manually labeled True and False identifiers. 55%~ of which are 1 (it's an identifier) and the remaining False

With the above setup, this are the results:

original strings: ["123/2111/0gg" , "644160949" , "B2921113", "27/04/1997", "foobar"]
replaced strings: ['ccceccccecaa', 'tcccccccc', 'bccccccc', 'cceccecccc', 'aaaaa']
Expected result: ['1', '0', '0', '0', '0']
Actual result:   ['1', '1', '1', '1', '0']

Since this approach doesn't seem to work, I'm kind of at a loss. Given that in the end it's binary text classification, what approach should I follow next? I imagine that it's hard to classify that kind of strings, but I'm ok even with a 65 - 75% of accuracy.

I've also tried multinomial naive bayes and SVC.


Working on a binary text classifier (either True or False) which aims to discriminate whether a string is an identifier or not.

I got 93% accuracy with Naive Bayes and CounterVectorizer but a lot of false positives comes up. I've already tried SVC and the Dataset contains ~40k rows of True strings and ~35k rows of False cases manually labelled.

Hardware is not an issue.

Can you give me any recommendations? Any kind of approach?

Much appreciated.

Edit just to clarify: I'm not really asking for a "solution", but rather whether my workaround is going the right way or if I should look for another method.

I understand why there are false positives popping up, and the reason behind its accuracy being so high... That's why the question I put at the end rather than being something along the lines of "why isnt this working...?" is asking for a recommendation or an approach to follow up.

BTW. Both the training and testing datasets have as close as it can get to 50% relationship between the two classes.