3
$\begingroup$

I am working on a binary classification problem. I have split the train set and when I evaluate the model on the validation set all metrics are 100% which is unrealistic considering that I haven't tuned the model. There are 14 features which are not including the labels. I have tried to change the hyperparameters of the RandomForestClassifier or tried others classifiers (DecisionTree, SVC) and I always get 100%. I know that this could be a sign of overfitting. Can anybody help me understanding what is causing this unexpected good results? Thank you in advance.

The features are based on the property of a twitter account you can get from twitter api. I've normalised some numerical features and changed objects to boolean, for instance screen_name -> has_digits (does the screen name includes digits?).


['default_profile', 'default_profile_image', 'geo_enabled',
       'profile_use_background_image', 'profile_background_tile', 'has_name',
       'has_digits', 'account_age', 'account_update_age', 'has_description',
'status_count_log', 'followers_count_log', 'friends_count_log',
       'favourites_count_log']


X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.2, random_state=42)


print(X_train.shape)
print(X_test.shape)
print(y_train.shape)
print(y_test.shape)

(5460, 14)
(1365, 14)
(5460,)
(1365,)

print('y_train class distribution')
print(y_train.value_counts(normalize=True))
print('y_test class distribution')
print(y_test.value_counts(normalize=True))

y_train class distribution
0    0.51337
1    0.48663

y_test class distribution
1    0.508425
0    0.491575

model = RandomForestClassifier(max_depth=28, min_samples_leaf=2, random_state=42)
model.fit(X_train, y_train)

y_test_pred = model.predict(X_test)
print('validation set:', roc_auc_score(y_test, y_test_pred)

validation set: 1.0

print(classification_report(y_test, y_test_pred))

            precision    recall  f1-score   support

           0       1.00      1.00      1.00       671
           1       1.00      1.00      1.00       694

   micro avg       1.00      1.00      1.00      1365
   macro avg       1.00      1.00      1.00      1365
weighted avg       1.00      1.00      1.00      1365


confusion_matrix(y_test, y_test_pred)

array([[671,   0],
       [  0, 694]])
$\endgroup$
  • 2
    $\begingroup$ I had a brief look but failed to find anything. Maybe your data was quite simple? $\endgroup$ – SmallChess Oct 3 '18 at 6:45
  • 1
    $\begingroup$ Are you sure you didn't leak target into features? It often caues such problems. Also did you try decreasing random forest's max_depth? $\endgroup$ – Jakub Bartczuk Oct 3 '18 at 9:27
  • $\begingroup$ I tried max_depth = 5 and got the same results. Before splitting I dropped the targets: features = df.drop(['target'], axis=1) $\endgroup$ – sergio Oct 3 '18 at 9:31
  • $\begingroup$ Without seeing the data it's going to be hard to answer. What makes you think you don't have one really good feature? $\endgroup$ – Calimo Oct 3 '18 at 11:59
  • $\begingroup$ I've added the features i'm using in the question above. I find difficult to believe that the good results are accurate, if I remove a lot of features I still get 100%, only if I have 2/3 features it drops to 50% $\endgroup$ – sergio Oct 3 '18 at 12:40
2
$\begingroup$

Check the feature importances to find out which feature(s) give this score. Quite likely there is some leakage of target into the features, or another form of leakage. Like having the index included as feature and classes appearing in a sorted order. This will show up as a column with feature importance approxing 100%

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ It doesn't look like 1 feature but a combination of features, is that possible? account_update_age 0.378430, status_count_log 0.217008 favourites_count_log 0.134432 followers_count_log 0.120690 default_profile 0.073779 geo_enabled 0.032114 profile_background_tile 0.017968 account_age 0.013488 friends_count_log 0.011164 profile_use_background_image 0.000616 default_profile_image 0.000203 has_digits 0.000072 has_description 0.000034 has_name 0.000000 $\endgroup$ – sergio Oct 3 '18 at 16:16
  • $\begingroup$ Those results look quite legitimate. Now you need to go and analyze the data in those first columns to understand how they together can give perfect predictions. $\endgroup$ – jonnor Oct 3 '18 at 17:55
  • $\begingroup$ You can try limiting the model to just the 3-5 best features. And set a limit on tree size. Might be that it reduces to something where you can look at the decisions themselves. $\endgroup$ – jonnor Oct 3 '18 at 18:19
  • $\begingroup$ It might be useful to see if the target is, in fact, completely determined by a subset of the features. You can do this by doing something like: $\endgroup$ – roundsquare Oct 6 '19 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.