I have a data set with 80 features of 250 observations (rows) (but the data set has missing values) . I want to do a binary classification for this data set.

I did feature selection using random forest and selected around 15 features (of 150 non missing observations) . Based on these features i fitted a generalized additive (GAM) logistic regression model . I obtained a training error of around 0.16 and a leave one out cross validation(LOOCV) error (test error) of 0.18. This result suggests that this model might under fit the data.

After that I ran both random forest and gradient boosting models for same data . (subset of 15 features) Based on the random forest model the training error is zero and the test error (LOOCV) is around 0.27. Similar kind of results obtained based on gradient boosting model too. Based on this two models, it seems that these two models over fit the data.

In all these models I defined error as the proportion of miss classification.



I want to know why these different models give contradictory (in terms of over fit and under fit) results and any suggestion that improve the results.



Thank you.