I want to do a multi class classification with 6 classes. Whole dataset has 12750 and 56 features samples, so every class has 2125 samples. Before prediction I reduces amount of outliers by winsorization (for 1 and 99 percentile) and I reduced skewness in features which has more than 1 and less than -1 skewness by Yeo-Johnson transformation and I got dataset:


Later, of course, I splitted dataset for 80% of training data and 20% of test data and I standardised training data. I tried to use random forest, xgboost and decision tree classifiers, but I have almost 100% accuracy on training set and 20-21% accuracy on test set. Methods like increasing n_estimators doesn't help.

So, my questions are:

How can I reduce this overfitting? Is it a problem with dataset (Should I reduce number of features something like that?) or with classificators (Are they too weak for this problem?)

Is the dataset too small for this problem (Should I add more samples by method like SMOTE?)? Do classes have too less samples to good work?

Is it possible to get at least 60% accuracy after tuning hyperparameters (e.g. by method like GridSearchCV)?

P.S. I will add that correlations with target value are very poor (max +- 6%) and I see that feature importances from random forest have values from 0.0 to 0.03. I don't know if this is a normal situation.

P.S.2 I tried to change n_estimators parameters (values from 5 to 1500) and max_depth (from 1 to 100) and I can see very poor change in test accuracy (+-3%)

  • $\begingroup$ Are you tuning the models' hyperparameters? How? Are you using holdout data to do the tuning? Please edit to clarify. $\endgroup$
    – Sycorax
    Commented Apr 25, 2022 at 20:42
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Commented Apr 25, 2022 at 20:52

1 Answer 1


I see that you've edited to state that you're tuning n_estimators and max_depth. For random forest, the most important parameter to tune is max_features (in python's sklearn) or mtry (in R's randomForest), the number of variables to consider for finding the best split. Moreover, you don't have to tune the number of trees in random forest; see: Do we have to tune the number of trees in a random forest?

For xgboost, the most important tuning parameter is the learning rate.

Finally, it's not clear how you're choosing hyperparameters. You should choose the best hyperparameters according to a holdout set, not the data used to train the model. See:

Is it possible to get at least 60% accuracy after tuning hyper-parameters (e.g. by method like GridSearchCV)?

The only way to know this is to do an experiment: tune the hyper-parameters and see. Some problems are harder than others. For instance, the features you have may not be very predictive of the outcome, so achieving the desired accuracy may not be possible.

  • $\begingroup$ Thank you for your answer. Can you explain how to do tuning hyperparameters according to a holdout set? I tried to choose the best parameters just giving them to the model before fitting, but after your answer I suppose this is wrong. $\endgroup$
    – jared
    Commented Apr 25, 2022 at 21:41
  • $\begingroup$ @jared I've edited my answer to include some links that elaborate. You can find many more using a search. Here's some tips for using search: stats.meta.stackexchange.com/questions/5549/… $\endgroup$
    – Sycorax
    Commented Apr 25, 2022 at 21:48
  • $\begingroup$ @jared I've edited my answer to reflect the additional question you edited into your post. $\endgroup$
    – Sycorax
    Commented Apr 26, 2022 at 12:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.