In a hypothetical case where I have a small dataset and I break it into train/test. Then I tune the hyperparams doing k-fold on the train set and choose the 'C' hyperparameter that maximizes my AUC on the validation set (kth fold). Let's say that the values of 'C' I try are 0.01, 0.1, 1, 10 and the gridCV chooses 0.1 for example. Would it make sense to manually lower this value by half (0.05) or even to 0.01 to penalize more the model since the odds that it is choosing a good hyperparameter with such a small dataset are very low?

What I want is for my model to generalize correctly to new cases and I have the impression that if I penalize my model more I will combat overfitting and it will generalize better.

For this hypothetical case, let's say that the mean validation AUC was 0.9 and my dataset only contains 100 patients and I did an 80/20 split for train/test. Also, I have around 15 features.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.