I'm performing NER (Named entity recognition)

For example:

Seq: When   Donald   Trump    announced...
Tags: O    B-Person L-Person     O 

When I'm predicting the word Trump, I have 'word features' for the word 'Trump' which are also considering the context, but I want to use the PREDICTED LABEL of the last word.

It means that I want to use the last predicted tag (Hopefully B-Person) as a feature when I'm predicting the word 'Trump'.

I understood there exists some way to do it using sklearn. How can I do it?



Why don't you use spacy (it's also the industry standard), it's super easy to do NER [https://spacy.io/usage/examples#training-ner][1] Then you can have all the information like this:

# Expected output:
# Entities [('Shaka Khan', 'PERSON')]
# Tokens [('Who', '', 2), ('is', '', 2), ('Shaka', 'PERSON', 3),
# ('Khan', 'PERSON', 1), ('?', '', 2)]
# Entities [('London', 'LOC'), ('Berlin', 'LOC')]
# Tokens [('I', '', 2), ('like', '', 2), ('London', 'LOC', 3),
# ('and', '', 2), ('Berlin', 'LOC', 3), ('.', '', 2)]

About the language (Hebrew), it's relatively simple to use it in spacy:

from spacy.he import Hebrew

tokenizer = Hebrew().tokenizer

print(list(w.text for w in tokenizer('עקבת אחריו בכל רחבי המדינה.')))
#  ['עקבת', 'אחריו', 'בכל', 'רחבי', 'המדינה.']
  • $\begingroup$ I'm doing NER on hebrew, not english $\endgroup$ – JohnSnowTheDeveloper Jan 18 '19 at 20:52
  • $\begingroup$ spacy support Hebrew (lang/he) $\endgroup$ – Marco Visibelli Jan 19 '19 at 18:09
  • $\begingroup$ Thanks, but I need to model it myself. Spacy is not relevant here. . This is a general machine learning question is about - How to use last predicted tag as feature. $\endgroup$ – JohnSnowTheDeveloper Jan 19 '19 at 19:55

I've used the naive approach:

  1. normal train & prediction.
  2. Getting the predicted previous tags for train and test using the trained classifier
  3. Creating new classifier and training it on the new data (the data with previous tags)
  4. Loop when predicting X_test[i], getting its tag and adding it as a feature to the next X_test[i+1].

    clf.fit(X_train, y_train)
    y_pred = clf.pred(X_test)
    y_train_prediction = clf.pred(X_train)
    X_train_with_tag = deepcopy(X_train)
    X_train_with_tag['prev_tag'] = ['O'] + y_train[:-1]
    clf_with_tag <- new clf 
    clf_with_tag.fit(X_train_with_tag, y_train)
    X_test[0]['prev_tag'] = 'O'
    new_y_pred = []
    for i in range(0, len(X_test)-1):
        pred = clf_model_with_tags.predict(X_test=X_test[i])
        X_test[i+1]['prev_tag'] = pred
    y_pred_after_tags = clf_with_tag.pred(X_test_with_tag)
    evaluate(y_test, y_pred_after_tags)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.