0
$\begingroup$

I am a bit confused with understanding the parameter step of RFE and RFECV algorithms. This is how I run RFECV for multi-class classification problem (3 classes):

svc = svm.SVC(kernel="linear", class_weight="balanced")
rfecv = feature_selection.RFECV(estimator=svc, step=50, 
                                cv=model_selection.StratifiedKFold(2), 
                                scoring='f1_weighted', n_jobs = -1,
                                verbose = 2)
rfecv.fit(X_train, y_train)

If I understand correctly, the algorithm starts with all features (in my case 681 features) and then it eliminates 50 features at each iteration (because step is equal to 50).

The optimal number of features for my dataset is equal to 631:

rfecv.n_features_

Then I plot the outputs as follows:

plt.figure()
plt.xlabel("Number of features selected")
plt.ylabel("Cross validation score (nb of correct classifications)")
plt.plot([int(x) for x in np.linspace(start = 681, stop = 0, num = 15)], rfecv.grid_scores_)
plt.show()

And get this chart:

enter image description here

Based on this image I would say that the best number of features is around 300-350.

If I start counting from 0 features, then I get another chart:

plt.figure()
plt.xlabel("Number of features selected")
plt.ylabel("Cross validation score (nb of correct classifications)")
plt.plot([int(x) for x in np.linspace(start = 0, stop = 681, num = 15)], rfecv.grid_scores_)
plt.show()

In this case indeed the optimal number of features is around 631, but I started counting from 0, not from 681!!! I analysed the source code of RFE and RFECV algorithms. Indeed it looks like the counting starts from all features, i.e. 681 features. So, why do I get wrong chart if I use [int(x) for x in np.linspace(start = 681, stop = 0, num = 15)]?

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.