3
$\begingroup$

I am working on a binary classification problem. I have split the train set and when I evaluate the model on the validation set all metrics are 100% which is unrealistic considering that I haven't tuned the model. There are 14 features which are not including the labels. I have tried to change the hyperparameters of the RandomForestClassifier or tried others classifiers (DecisionTree, SVC) and I always get 100%. I know that this could be a sign of overfitting. Can anybody help me understanding what is causing this unexpected good results? Thank you in advance.

The features are based on the property of a twitter account you can get from twitter api. I've normalised some numerical features and changed objects to boolean, for instance screen_name -> has_digits (does the screen name includes digits?).


['default_profile', 'default_profile_image', 'geo_enabled',
       'profile_use_background_image', 'profile_background_tile', 'has_name',
       'has_digits', 'account_age', 'account_update_age', 'has_description',
'status_count_log', 'followers_count_log', 'friends_count_log',
       'favourites_count_log']


X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.2, random_state=42)


print(X_train.shape)
print(X_test.shape)
print(y_train.shape)
print(y_test.shape)

(5460, 14)
(1365, 14)
(5460,)
(1365,)

print('y_train class distribution')
print(y_train.value_counts(normalize=True))
print('y_test class distribution')
print(y_test.value_counts(normalize=True))

y_train class distribution
0    0.51337
1    0.48663

y_test class distribution
1    0.508425
0    0.491575

model = RandomForestClassifier(max_depth=28, min_samples_leaf=2, random_state=42)
model.fit(X_train, y_train)

y_test_pred = model.predict(X_test)
print('validation set:', roc_auc_score(y_test, y_test_pred)

validation set: 1.0

print(classification_report(y_test, y_test_pred))

            precision    recall  f1-score   support

           0       1.00      1.00      1.00       671
           1       1.00      1.00      1.00       694

   micro avg       1.00      1.00      1.00      1365
   macro avg       1.00      1.00      1.00      1365
weighted avg       1.00      1.00      1.00      1365


confusion_matrix(y_test, y_test_pred)

array([[671,   0],
       [  0, 694]])
Improve this question
$\endgroup$
10
  • 2
    $\begingroup$ I had a brief look but failed to find anything. Maybe your data was quite simple? $\endgroup$
    – SmallChess
    Commented Oct 3, 2018 at 6:45
  • 1
    $\begingroup$ Are you sure you didn't leak target into features? It often caues such problems. Also did you try decreasing random forest's max_depth? $\endgroup$
    – Jakub Bartczuk
    Commented Oct 3, 2018 at 9:27
  • $\begingroup$ I tried max_depth = 5 and got the same results. Before splitting I dropped the targets: features = df.drop(['target'], axis=1) $\endgroup$
    – sergio
    Commented Oct 3, 2018 at 9:31
  • $\begingroup$ Without seeing the data it's going to be hard to answer. What makes you think you don't have one really good feature? $\endgroup$
    – Calimo
    Commented Oct 3, 2018 at 11:59
  • $\begingroup$ I've added the features i'm using in the question above. I find difficult to believe that the good results are accurate, if I remove a lot of features I still get 100%, only if I have 2/3 features it drops to 50% $\endgroup$
    – sergio
    Commented Oct 3, 2018 at 12:40

1 Answer 1

Reset to default
2
$\begingroup$

Check the feature importances to find out which feature(s) give this score. Quite likely there is some leakage of target into the features, or another form of leakage. Like having the index included as feature and classes appearing in a sorted order. This will show up as a column with feature importance approxing 100%

Improve this answer
$\endgroup$
4
  • $\begingroup$ It doesn't look like 1 feature but a combination of features, is that possible? account_update_age 0.378430, status_count_log 0.217008 favourites_count_log 0.134432 followers_count_log 0.120690 default_profile 0.073779 geo_enabled 0.032114 profile_background_tile 0.017968 account_age 0.013488 friends_count_log 0.011164 profile_use_background_image 0.000616 default_profile_image 0.000203 has_digits 0.000072 has_description 0.000034 has_name 0.000000 $\endgroup$
    – sergio
    Commented Oct 3, 2018 at 16:16
  • $\begingroup$ Those results look quite legitimate. Now you need to go and analyze the data in those first columns to understand how they together can give perfect predictions. $\endgroup$
    – Jon Nordby
    Commented Oct 3, 2018 at 17:55
  • $\begingroup$ You can try limiting the model to just the 3-5 best features. And set a limit on tree size. Might be that it reduces to something where you can look at the decisions themselves. $\endgroup$
    – Jon Nordby
    Commented Oct 3, 2018 at 18:19
  • $\begingroup$ It might be useful to see if the target is, in fact, completely determined by a subset of the features. You can do this by doing something like: $\endgroup$
    – roundsquare
    Commented Oct 6, 2019 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.