# How to interpret ROC curve with confusion matrix and F1 score?

I have implemented a random forest classifier to do a binary classification in highly imbalanced class. As the performance measurements, ROC and the f1 score was considered. However, the ROC curve gives AUC of 1 which corresponding to the perfect value and I am a little bit worried about it since it is the perfect solution. I have attached the ROC curve confusion matrix below and the F1 score is 0.99942. Can someone interpret these results to check if the model is good or bad?

dfinput =pd.read_csv(os.path.join(os.path.abspath(os.path.dirname(__file__)), "../data.csv"))
dfoutput = dfinput["target"]
dfinput = dfinput[selecting feature columns]
X = np.array(dfinput)
y = np.array(dfoutput)

scaler = StandardScaler()
X = scaler.transform(X)
randomforestclassifier = RandomForestClassifier(n_estimators=50)

print("Feature Importance Started")
model = ExtraTreesClassifier()
model.fit(X, y)
print(model.feature_importances_)
print("Feature Importance Ended")

X_train, X_test, y_train,y_test = train_test_split(X,y,random_state=3)
X_resampled, y_resampled = SMOTE(ratio='minority').fit_sample(X_train, y_train)
randomforestclassifier.fit(X_resampled, y_resampled)
list1 = randomforestclassifier.predict(X_test)
list2 = randomforestclassifier.predict(X_train)

prob_test = randomforestclassifier.predict_proba(X_test)
predictions = pd.DataFrame(prob_test)
predictions = predictions.iloc[:, 1]

false_positive_rate, true_positive_rate, thresholds = roc_curve(y_test, predictions)
roc_auc = auc(false_positive_rate, true_positive_rate)

plt.title('Receiver Operating Characteristic of Test set')
plt.plot(false_positive_rate, true_positive_rate, 'b',
label='AUC = %0.2f'% roc_auc)
plt.legend(loc='lower right')
plt.plot([0,1],[0,1],'r--')
plt.xlim([-0.1,1.2])
plt.ylim([-0.1,1.2])
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()

print("F Score for test data %s "%(f1_score(y_test, list1, average='micro')  ))


Confusion matrix

ROC curve

• You may be Neo from the matrix: the chosen one who has solved a problem almost perfectly. However, from my experience, I can only say that I understand your worries and highly doubt that. Without knowing anything further about the code or the data my guesses are: 1) you included an 'illegal' variable that is either the answer itself or something close to the true answer. 2) there is some problem in the evaluation logic/grouping/... – Fabian Werner Aug 26 '18 at 10:37
• @FabianWerner I added the source code. Also when I applied cross-validation it gave good results as well. – Sidath Asiri Aug 26 '18 at 10:56
• One plot is missing: what does the feature importance look like? This might help in figuring out which feature might be the faulty one... – Fabian Werner Aug 26 '18 at 10:58
• [7.26015980e-02 7.25199703e-02 5.44723388e-02 1.11711937e-01 1.04243051e-01 1.20475797e-01 1.52748141e-01 9.90904056e-02 2.12067774e-01 6.89864711e-05] these are the feature imporance values for each feature. The code of confusion matrix is directly coming from sklearn – Sidath Asiri Aug 26 '18 at 11:03
• That is good but which feature corresponds to which value? – Fabian Werner Aug 26 '18 at 12:16