1
$\begingroup$

I am working to improve classification results with more ML algorithm. I get 100 percent accuracy in both test and training set.

I used GradientBoostingClassifier, XGboost , RandomForest and Xgboost with GridSearchCV

My daset shape is (222,70), for the 70 features i have 25 binary features and 44 continious features.

My dataset looks like this:

enter image description here

My code for Xgboost with GridSearchCV looks like this :

from sklearn.model_selection import GridSearchCV
import xgboost as xgb

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15,random_state=141)
steps=[('scaler', StandardScaler()), 
       ('XGB',xgb.XGBClassifier(objective= 'binary:logistic'))]
                                                          
from sklearn.pipeline import Pipeline
pipeline = Pipeline(steps) # define the pipeline object.

parameteres = {'XGB__learning_rate':[0.01,0.1, 0.25,0.5,0.7,0.8,1], 
               'XGB__n_estimators':[5,8,9,10,11,20,100],
               'XGB__max_depth':[1,2,3,5,8,10,20]}

grid = GridSearchCV(pipeline, param_grid=parameteres, cv=5,verbose=True)
grid.fit(X_train, y_train)
print ("score = %3.2f" %(grid.score(X_test,y_test)))
print (grid.best_params_)

The result looks like this :

score = 1.00
{'XGB__learning_rate': 0.01, 'XGB__max_depth': 1, 'XGB__n_estimators': 5}

And the result with GradientBoostingClassifier and RandomForest looks like this

GradientBoostingClassifier
****Results****
Accuracy test: 100.0000%
Accuracy trainning: 100.0000%
Log Loss: 9.992007221626415e-16
confusion matrix
matrix: [[18  0]
        [ 0 16]]

RandomForestClassifier
****Results****
Accuracy test: 100.0000%
Accuracy trainning: 100.0000%
Log Loss: 9.992007221626415e-16
confusion matrix
matrix: [[18  0]
        [ 0 16]]

How to interpret this result ?

It is possible that my classifiers are overfitting the training set ?

$\endgroup$
  • $\begingroup$ I don't program in R, so I can't help you with the code, but if you had an overfitting problem the accuracy wouldn't be 100% also in the test set. However, it's very unlikely that the accuracy is actually perfect in both sets... $\endgroup$ – Johanna Aug 27 at 15:23
  • $\begingroup$ Thank you. The code is in Python. i think that there is a problem with my dataset $\endgroup$ – Dhayf OTHMEN Aug 27 at 15:35
  • $\begingroup$ If you have a depth of 1 and are getting perfect accuracy then you probably have significant leakage. Looks like your target variable is probably in your X. $\endgroup$ – Tylerr Aug 28 at 13:36
  • $\begingroup$ @Tylerr what you means by "depth of 1" please ? $\endgroup$ – Dhayf OTHMEN Aug 28 at 14:27
  • $\begingroup$ @dhifallahothmen Looks like your gridsearch returned this as the optimal parameters: {'XGB__learning_rate': 0.01, 'XGB__max_depth': 1, 'XGB__n_estimators': 5} which is a very very simple model. Your tree only makes one split each iteration (this is the depth) and I bet you could get that 100% accuracy with XGB_n_estimators set to 1 because you have something in your X matrix which is 100% correlated with your y and it most likely IS your y or some metric which is based on your y. Basically, you have bad data leakage somewhere. $\endgroup$ – Tylerr Aug 28 at 14:54