Different machine learning models give contradictory results

Question

I am relatively new to data analysis using machine learning and i got following problem while doing this machine learning problem.

I have a data set with 80 features of 250 observations (rows) (but the data set has missing values) . I want to do a binary classification for this data set.

I did feature selection using random forest and selected around 15 features (of 150 non missing observations) . Based on these features i fitted a generalized additive (GAM) logistic regression model . I obtained a training error of around 0.16 and a leave one out cross validation(LOOCV) error (test error) of 0.18. This result suggests that this model might under fit the data.

After that I ran both random forest and gradient boosting models for same data . (subset of 15 features) Based on the random forest model the training error is zero and the test error (LOOCV) is around 0.27. Similar kind of results obtained based on gradient boosting model too. Based on this two models, it seems that these two models over fit the data.

In all these models I defined error as the proportion of miss classification.

I want to know why these different models give contradictory (in terms of over fit and under fit) results and any suggestion that improve the results.

Any suggestion would be highly appreciated .

Thank you.

Answer 1

First of all I would recomment this book here to you: Hands-On Machine Learning with Scikit-Learn and TensorFlow

There you can read about overfitting, underfitting and the reasons.

You can imagine overfitting as a process where the model remembers the training data to detailed and can not generalize to the test or validation data. In summary you can say that "simple" models tend to underfit. Because they can not "memorise" so well. Complex and bigger models tend to overfit. To fight overfitting you need regularisation or / and early stopping.

In general you want to find a model that overfits (or increase the potential of the model so it overfits) and then apply regularisation.

For gradient boosted trees I love to use lightGBM. Its awsome. Here you have so called hyperparameters (many) that can be canged to fight overfitting. Gradient Boosted trees can handle missing values and there is no need to feature select. And no need for feature scaling.

Here is an example of lightGBM usage. Maybe you want to try this with your own data: https://github.com/PhilipMay/fraud-detection/blob/master/LightGBM_with_Hyperopt_no_oversampling_0.896.ipynb

Deal with overfitting in lightGBM: https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html

By the way: In my experience lightGBM is one of the best tools to do classification or regression with plain data. Better then neural networks. Neural Networks are better when it comes to Images, Time Series or sequence data or NLP. And lightGBM is like 1000 times faster then neural networks.