Low classification accuracy

Question

I want to do a multi class classification with 6 classes. Whole dataset has 12750 and 56 features samples, so every class has 2125 samples. Before prediction I reduces amount of outliers by winsorization (for 1 and 99 percentile) and I reduced skewness in features which has more than 1 and less than -1 skewness by Yeo-Johnson transformation and I got dataset:

https://i.sstatic.net/miy8i.png

Later, of course, I splitted dataset for 80% of training data and 20% of test data and I standardised training data. I tried to use random forest, xgboost and decision tree classifiers, but I have almost 100% accuracy on training set and 20-21% accuracy on test set. Methods like increasing n_estimators doesn't help.

So, my questions are:

How can I reduce this overfitting? Is it a problem with dataset (Should I reduce number of features something like that?) or with classificators (Are they too weak for this problem?)

Is the dataset too small for this problem (Should I add more samples by method like SMOTE?)? Do classes have too less samples to good work?

Is it possible to get at least 60% accuracy after tuning hyperparameters (e.g. by method like GridSearchCV)?

P.S. I will add that correlations with target value are very poor (max +- 6%) and I see that feature importances from random forest have values from 0.0 to 0.03. I don't know if this is a normal situation.

P.S.2 I tried to change n_estimators parameters (values from 5 to 1500) and max_depth (from 1 to 100) and I can see very poor change in test accuracy (+-3%)

Answer 1

I see that you've edited to state that you're tuning n_estimators and max_depth. For random forest, the most important parameter to tune is max_features (in python's sklearn) or mtry (in R's randomForest), the number of variables to consider for finding the best split. Moreover, you don't have to tune the number of trees in random forest; see: Do we have to tune the number of trees in a random forest?

For xgboost, the most important tuning parameter is the learning rate.

Finally, it's not clear how you're choosing hyperparameters. You should choose the best hyperparameters according to a holdout set, not the data used to train the model. See:

• How bad is hyperparameter tuning outside cross-validation?

• Step-by-step explanation of K-fold cross-validation with grid search to optimise hyperparameters

• What is the difference between test set and validation set?

Is it possible to get at least 60% accuracy after tuning hyper-parameters (e.g. by method like GridSearchCV)?

The only way to know this is to do an experiment: tune the hyper-parameters and see. Some problems are harder than others. For instance, the features you have may not be very predictive of the outcome, so achieving the desired accuracy may not be possible.