I have implemented a random forest classifier to do a binary classification in highly imbalanced class. As the performance measurements, ROC and the f1 score was considered. However, the ROC curve gives AUC of 1 which corresponding to the perfect value and I am a little bit worried about it since it is the perfect solution. I have attached the ROC curve confusion matrix below and the F1 score is 0.99942. Can someone interpret these results to check if the model is good or bad?

dfinput =pd.read_csv(os.path.join(os.path.abspath(os.path.dirname(__file__)), "../data.csv"))
dfoutput = dfinput["target"]
dfinput = dfinput[selecting feature columns]
X = np.array(dfinput)
y = np.array(dfoutput)

scaler = StandardScaler()
X = scaler.transform(X)
randomforestclassifier = RandomForestClassifier(n_estimators=50)

print("Feature Importance Started")
model = ExtraTreesClassifier()
model.fit(X, y)
print("Feature Importance Ended")

X_train, X_test, y_train,y_test = train_test_split(X,y,random_state=3)
X_resampled, y_resampled = SMOTE(ratio='minority').fit_sample(X_train, y_train)
randomforestclassifier.fit(X_resampled, y_resampled)
list1 = randomforestclassifier.predict(X_test)
list2 = randomforestclassifier.predict(X_train)

prob_test = randomforestclassifier.predict_proba(X_test)
predictions = pd.DataFrame(prob_test)
predictions = predictions.iloc[:, 1]

false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, predictions)
roc_auc = auc(false_positive_rate, true_positive_rate)

plt.title('Receiver Operating Characteristic of Test set')
plt.plot(false_positive_rate, true_positive_rate, 'b',
label='AUC = %0.2f'% roc_auc)
plt.legend(loc='lower right')
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')

print("F Score for test data %s "%(f1_score(y_test, list1, average='micro')  ))

Confusion matrix Confusion matrix

ROC curve ROC curve

  • $\begingroup$ You may be Neo from the matrix: the chosen one who has solved a problem almost perfectly. However, from my experience, I can only say that I understand your worries and highly doubt that. Without knowing anything further about the code or the data my guesses are: 1) you included an 'illegal' variable that is either the answer itself or something close to the true answer. 2) there is some problem in the evaluation logic/grouping/... $\endgroup$ – Fabian Werner Aug 26 '18 at 10:37
  • $\begingroup$ @FabianWerner I added the source code. Also when I applied cross-validation it gave good results as well. $\endgroup$ – Sidath Asiri Aug 26 '18 at 10:56
  • $\begingroup$ One plot is missing: what does the feature importance look like? This might help in figuring out which feature might be the faulty one... $\endgroup$ – Fabian Werner Aug 26 '18 at 10:58
  • $\begingroup$ [7.26015980e-02 7.25199703e-02 5.44723388e-02 1.11711937e-01 1.04243051e-01 1.20475797e-01 1.52748141e-01 9.90904056e-02 2.12067774e-01 6.89864711e-05] these are the feature imporance values for each feature. The code of confusion matrix is directly coming from sklearn $\endgroup$ – Sidath Asiri Aug 26 '18 at 11:03
  • $\begingroup$ That is good but which feature corresponds to which value? $\endgroup$ – Fabian Werner Aug 26 '18 at 12:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.