0
$\begingroup$

I am trying to perform feature interpretability on a text corpus but I am becoming quite confused as to how I identify the importance of particular features (words). I have done substantial research to try and figure this out and have provided an implementation below but for some reason it is not coming out the way I would expect. I'd appreciate direction as to whether I have the wrong theory or if I have a bug in my code.

Theory: From what I understand obtaining feature interpretability is easiest when using a linear model. To that end I have trained an SVM with a linear kernel, and a one vs. rest approach for multiclass learning. According to this post I should be able to obtain the individual feature significance from the coef_ variable of the trained SVM. When parsing my text through either a TF-IDF or Count Vectorizer and I receive the same output (so I am mostly certain this is not related to the issue).

My text corpus is simple, and involves several of the same lines with changes only to a single word. The same is true of the evaluation corpus. As such a classification can only be made using the words: 'cat', 'dog', and 'rat'.

Code:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn import svm, metrics

import numpy as np

# Setup training/eval data

Xtrain = [
    "I am a cat",
    "What do i cat",
    "a cat is where its at",
    "how about that cat",
    "the cat there sat",

    "I am a dog",
    "What do i dog",
    "a dog is where its at",
    "how about that dog",
    "the dog there sat",

    "I am a rat",
    "What do i rat",
    "a rat is where its at",
    "how about that rat",
    "the rat there sat",]

Ytrain = [0,0,0,0,0, 1,1,1,1,1, 2,2,2,2,2]

Xeval = [
    "it was a cat",
    "it was a dog",
    "it was a rat",

    "I watched cat where it sat",
    "I watched dog where it sat",
    "I watched rat where it sat",]

Yeval = [0,1,2,0,1,2]


# define model

vect = CountVectorizer()
clf = svm.SVC(max_iter=100, tol=1e-4, probability=True, 
kernel='linear', decision_function_shape='ovr' )

Xtrain_vect = vect.fit_transform(Xtrain)
clf.fit(Xtrain_vect, Ytrain)

# fit model
pred = clf.predict(vect.transform(Xeval))
print(metrics.accuracy_score(pred, Yeval))   # Achieves 100% accuracy

#--------------------------
# understanding features
#--------------------------
fn = np.array(vect.get_feature_names())
print(fn)   # prints vocabulary

coef = clf.coef_.toarray()
print(coef.shape) # (3, 16)


# for each of the available classes I try to output the 
# most significant features to that model, because I use 
# a one vs. rest approach I anticipate that I should see
# 'cat', 'dog', and 'rat' as the most significant word
# in each of the three models.

for label in range(coef.shape[0]):

    print('')
    print("Label: ", label)

    cf = coef[label].reshape(-1)

    # sort features in descending order
    order = cf.argsort()
    cf = cf[order][::-1]
    fn = fn[order][::-1]

    # print feature importance 
    for f, c in zip(fn, cf):
        print(f, c)

Result: Because I am using a one vs. rest approach to my SVM I anticipate that 'cat', 'dog', and 'rat' should be the most important features in the prediction. This is especially true considering that model is 100% accurate on the evaluation dataset. However when I run the code I am seeing the following output:

('Label: ', 0)

(u'cat', 1.0)
(u'where', 0.0)
(u'what', 0.0)
    #condensed for space
(u'am', 0.0)
(u'about', 0.0)
(u'dog', -1.0)

('Label: ', 1)

(u'there', 1.0)
(u'dog', 0.0)
(u'about', 0.0)
     #condensed for space
(u'where', 0.0)
(u'cat', 0.0)
(u'is', -1.0)

('Label: ', 2)
(u'do', 1.0)
(u'is', 0.0)
(u'cat', 0.0)
     #condensed for space
(u'dog', 0.0)
(u'there', 0.0)
(u'sat', -1.0)

As can be observed, the first class correctly identifies 'cat' as being very important but only considers 'dog' to be uninformative. For the other two labels I am receiving words I would consider uninformative as the most important and least important words.

What am I doing wrong? Thank you in advance.

UPDATE: So I have discovered this is in part due to a bug, namely that I was not deep-copying the Feature name list and was manipulating it each time I modified the 'order' for each new label. This explains why the first label made some sense. When I correct this bug I am getting the following output:

('Label: ', 0)

(u'cat', 1.0)
(u'where', 0.0)
...
(u'about', 0.0)
(u'dog', -1.0)

('Label: ', 1)

(u'cat', 1.0)
(u'where', 0.0)
...
(u'about', 0.0)
(u'rat', -1.0)

('Label: ', 2)

(u'dog', 1.0)
(u'where', 0.0)
...
(u'about', 0.0)
(u'rat', -1.0)
$\endgroup$

1 Answer 1

0
$\begingroup$

I resolved this problem some time ago and thought I would present the answer for those who are looking.

In short, the Sklearn SVM does not perform in a typical one-vs-rest implementation. Instead it uses a wrapper to develop a decision function that combines the decision functions of several one-vs-other classifiers to perform its classification. This approach has changed between implementations and I encourage you to refer to your installation of Sklearn to find out how yours is being performed.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.