I am working on a binary classification problem. I have split the train set and when I evaluate the model on the validation set all metrics are 100% which is unrealistic considering that I haven't tuned the model. There are 14 features which are not including the labels. I have tried to change the hyperparameters of the RandomForestClassifier or tried others classifiers (DecisionTree, SVC) and I always get 100%. I know that this could be a sign of overfitting. Can anybody help me understanding what is causing this unexpected good results? Thank you in advance.
The features are based on the property of a twitter account you can get from twitter api. I've normalised some numerical features and changed objects to boolean, for instance screen_name -> has_digits (does the screen name includes digits?).
['default_profile', 'default_profile_image', 'geo_enabled', 'profile_use_background_image', 'profile_background_tile', 'has_name', 'has_digits', 'account_age', 'account_update_age', 'has_description', 'status_count_log', 'followers_count_log', 'friends_count_log', 'favourites_count_log']
X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.2, random_state=42) print(X_train.shape) print(X_test.shape) print(y_train.shape) print(y_test.shape) (5460, 14) (1365, 14) (5460,) (1365,) print('y_train class distribution') print(y_train.value_counts(normalize=True)) print('y_test class distribution') print(y_test.value_counts(normalize=True)) y_train class distribution 0 0.51337 1 0.48663 y_test class distribution 1 0.508425 0 0.491575 model = RandomForestClassifier(max_depth=28, min_samples_leaf=2, random_state=42) model.fit(X_train, y_train) y_test_pred = model.predict(X_test) print('validation set:', roc_auc_score(y_test, y_test_pred) validation set: 1.0 print(classification_report(y_test, y_test_pred)) precision recall f1-score support 0 1.00 1.00 1.00 671 1 1.00 1.00 1.00 694 micro avg 1.00 1.00 1.00 1365 macro avg 1.00 1.00 1.00 1365 weighted avg 1.00 1.00 1.00 1365 confusion_matrix(y_test, y_test_pred) array([[671, 0], [ 0, 694]])