Im using rbf svm classifier with nested cross validation (5 kfold to tune hyperparameters and then leave the last 10% for testing). When tuning hyperparameters the best cv accuracy trains to around 56%, but the training accuracy is 100%! The best cv score is found for relatively high values of gamma. The test set performs significantly worse than the cross validation score (around 50%). What should I do in this case? Obviously the model is overfit, but the overfit model returns the best cross validation score. How can this be alleviated?
Note: I have ~10k data points and ~50 features so I ont think small dataset is the problem.