I am still very new to machine learning and trying to figure things out myself. I am using SciKit learn and have a data set of tweets with around 20,000 features (n_features=20,000). So far I achieved a precision, recall and f1 score of around 79%. I would like to use RFECV for feature selection and improve the performance of my model. I have read the SciKit learn documentation but am still a bit confused on how to use RFECV.
This is the code I have so far:
from sklearn.feature_extraction.text import CountVectorizer from sklearn.feature_extraction.text import TfidfTransformer from sklearn.naive_bayes import MultinomialNB from sklearn.cross_validation import StratifiedShuffleSplit from sklearn.cross_validation import cross_val_score from sklearn.feature_selection import RFECV from sklearn import metrics # cross validation sss = StratifiedShuffleSplit(y, 5, test_size=0.2, random_state=42) for train_index, test_index in sss: docs_train, docs_test = X[train_index], X[test_index] y_train, y_test = y[train_index], y[test_index] # feature extraction count_vect = CountVectorizer(stop_words='English', min_df=3, max_df=0.90, ngram_range=(1,3)) X_CV = count_vect.fit_transform(docs_train) tfidf_transformer = TfidfTransformer() X_tfidf = tfidf_transformer.fit_transform(X_CV) # Create the RFECV object nb = MultinomialNB(alpha=0.5) # The "accuracy" scoring is proportional to the number of correct classifications rfecv = RFECV(estimator=nb, step=1, cv=2, scoring='accuracy') rfecv.fit(X_tfidf, y_train) X_rfecv=rfecv.transform(X_tfidf) print("Optimal number of features : %d" % rfecv.n_features_) # train classifier clf = MultinomialNB(alpha=0.5).fit(X_rfecv, y_train) # test clf on test data X_test_CV = count_vect.transform(docs_test) X_test_tfidf = tfidf_transformer.transform(X_test_CV) X_test_rfecv = rfecv.transform(X_test_tfidf) y_predicted = clf.predict(X_test_rfecv) #print the mean accuracy on the given test data and labels print ("Classifier score is: %s " % rfecv.score(X_test_rfecv,y_test))
1) Is this the correct way to use cross validation and RFECV? I am especially interested to know if I am running any risk of overfitting.
2) The accuracy of my model before and after I implemented RFECV with the above code are almost the same (around 78-79%), which puzzles me. I would expect performance to improve by using RFECV. Anything I might have missed here or could do differently to improve the performance of my model?
3) What other feature selection methods could you recommend me to try? I have tried RFE and SelectKBest so far, but they both haven't given me any improvement in terms of model accuracy.