1
$\begingroup$

I am using 10-fold validation method to validate my model. I am using CART model and my sample size $\approx$ 50. Features $\approx$ 9. The 10-fold validated accuracy (averages) is about 76%. However, I noticed that there is a a huge variation in the accuracy % for each of the folds ranging from 30% - 100%. Is this normal?

Also, when I randomly permute the order of my observations and train my CART model (and repeat this process), there is a variation in my accuracy % (ranging from 68% - 84%). Right now I am doing about 25 trials and using the average from these 25 trials to gauge my model performance. Is this the right approach?

$\endgroup$

1 Answer 1

1
$\begingroup$

First of all, I do not know about CART models, so this is not a conclusive answer. I do have a decent background on cross-validation techniques and applied statistics in general and I can tell you what my intuition says about what is going on behind the scenes...

It seems to me that overfitting is causing that variability: you have $n\approx 50$ observations and you try to model a classification with a number of features around $9$, which probably means that the number of variables or tuning parameters is close or up to that figure. This is almost $5$ observations to get information for one feature and the risk of overfitting for me is evident. Furthermore, when you do $10$-fold cross-validation to test your trained model implies that you fit the model with something like $45$ data and you validate on $5$. I think that the high variability that you are getting is because prediction is too much focused on the training data and is ignoring the big picture of the data.

My advice to focus this problem:

1) Remove the less predictive features or

2) get more data (could be impossible, I know).

Hope this general answer helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.