I am still very new to machine learning and trying to figure things out myself. I am using SciKit learn and have a data set of tweets with around 20,000 features (n_features=20,000). So far I achieved a precision, recall and f1 score of around 79%. I would like to use RFECV for feature selection and improve the performance of my model. I have read the SciKit learn documentation but am still a bit confused on how to use RFECV.

This is the code I have so far:

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.naive_bayes import MultinomialNB
from sklearn.cross_validation import StratifiedShuffleSplit
from sklearn.cross_validation import cross_val_score
from sklearn.feature_selection import RFECV
from sklearn import metrics

# cross validation
sss = StratifiedShuffleSplit(y, 5, test_size=0.2, random_state=42)
for train_index, test_index in sss:
    docs_train, docs_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

# feature extraction
count_vect = CountVectorizer(stop_words='English', min_df=3, max_df=0.90, ngram_range=(1,3))
X_CV = count_vect.fit_transform(docs_train)

tfidf_transformer = TfidfTransformer()
X_tfidf = tfidf_transformer.fit_transform(X_CV)

# Create the RFECV object
nb = MultinomialNB(alpha=0.5)

# The "accuracy" scoring is proportional to the number of correct classifications
rfecv = RFECV(estimator=nb, step=1, cv=2, scoring='accuracy')

rfecv.fit(X_tfidf, y_train)

print("Optimal number of features : %d" % rfecv.n_features_)

# train classifier
clf = MultinomialNB(alpha=0.5).fit(X_rfecv, y_train)

# test clf on test data

X_test_CV = count_vect.transform(docs_test)
X_test_tfidf = tfidf_transformer.transform(X_test_CV)
X_test_rfecv = rfecv.transform(X_test_tfidf)

y_predicted = clf.predict(X_test_rfecv)

#print the mean accuracy on the given test data and labels

print ("Classifier score is: %s " % rfecv.score(X_test_rfecv,y_test))

Three questions:

1) Is this the correct way to use cross validation and RFECV? I am especially interested to know if I am running any risk of overfitting.

2) The accuracy of my model before and after I implemented RFECV with the above code are almost the same (around 78-79%), which puzzles me. I would expect performance to improve by using RFECV. Anything I might have missed here or could do differently to improve the performance of my model?

3) What other feature selection methods could you recommend me to try? I have tried RFE and SelectKBest so far, but they both haven't given me any improvement in terms of model accuracy.

  • 1
    $\begingroup$ Please spell out your acronyms. What is "RFECV"? Be aware the code check is off topic here. You may want to give a brief text summary of what your code does for people who don't use Python. $\endgroup$ – gung Aug 20 '15 at 5:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.