# Classification model accuracy, roc auc score, f1 score 100%

I am working on a binary classification problem. I have split the train set and when I evaluate the model on the validation set all metrics are 100% which is unrealistic considering that I haven't tuned the model. There are 14 features which are not including the labels. I have tried to change the hyperparameters of the RandomForestClassifier or tried others classifiers (DecisionTree, SVC) and I always get 100%. I know that this could be a sign of overfitting. Can anybody help me understanding what is causing this unexpected good results? Thank you in advance.

The features are based on the property of a twitter account you can get from twitter api. I've normalised some numerical features and changed objects to boolean, for instance screen_name -> has_digits (does the screen name includes digits?).


['default_profile', 'default_profile_image', 'geo_enabled',
'profile_use_background_image', 'profile_background_tile', 'has_name',
'has_digits', 'account_age', 'account_update_age', 'has_description',
'status_count_log', 'followers_count_log', 'friends_count_log',
'favourites_count_log']


X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.2, random_state=42)

print(X_train.shape)
print(X_test.shape)
print(y_train.shape)
print(y_test.shape)

(5460, 14)
(1365, 14)
(5460,)
(1365,)

print('y_train class distribution')
print(y_train.value_counts(normalize=True))
print('y_test class distribution')
print(y_test.value_counts(normalize=True))

y_train class distribution
0    0.51337
1    0.48663

y_test class distribution
1    0.508425
0    0.491575

model = RandomForestClassifier(max_depth=28, min_samples_leaf=2, random_state=42)
model.fit(X_train, y_train)

y_test_pred = model.predict(X_test)
print('validation set:', roc_auc_score(y_test, y_test_pred)

validation set: 1.0

print(classification_report(y_test, y_test_pred))

precision    recall  f1-score   support

0       1.00      1.00      1.00       671
1       1.00      1.00      1.00       694

micro avg       1.00      1.00      1.00      1365
macro avg       1.00      1.00      1.00      1365
weighted avg       1.00      1.00      1.00      1365

confusion_matrix(y_test, y_test_pred)

array([[671,   0],
[  0, 694]])

• I had a brief look but failed to find anything. Maybe your data was quite simple? – SmallChess Oct 3 '18 at 6:45
• Are you sure you didn't leak target into features? It often caues such problems. Also did you try decreasing random forest's max_depth? – Jakub Bartczuk Oct 3 '18 at 9:27
• I tried max_depth = 5 and got the same results. Before splitting I dropped the targets: features = df.drop(['target'], axis=1) – sergio Oct 3 '18 at 9:31
• Without seeing the data it's going to be hard to answer. What makes you think you don't have one really good feature? – Calimo Oct 3 '18 at 11:59
• I've added the features i'm using in the question above. I find difficult to believe that the good results are accurate, if I remove a lot of features I still get 100%, only if I have 2/3 features it drops to 50% – sergio Oct 3 '18 at 12:40