I am solving the problem of detecting swallowing and non-swallowing events from the audio. I labelled the data using Praat software by marking the swallowing and nonswallowing events. I trained the model using LibSVM with balanced dataset of 1841 instance of each and test with non-balanced dataset 369 non-swallow and 548 swallow events. I performed grid-search on optimal $(C,\gamma)$ and find them as $(2,0.25)$ which 5-fold cross validation accuracy as 90%, which is for total of 29 record. But when I tested the model for test data, I obtained 60% accuracy. I also tested for balanced data, but the result still not acceptable. If the model is not generalized or overfit, why cross-validation accuracy is 90% ? Any idea? How can I fix the problem?
2
-
$\begingroup$ How did you find this parameter combination? If you simply ran a single CV over the whole training set, that can surely result in overfit. Ideally, it should be done in a nested CV, where selecting the best hyperparameters become part of the model building process. $\endgroup$– FirebugCommented Mar 24, 2023 at 10:26
-
$\begingroup$ @Firebug Thanks for your comment. I used LibSVM cross-validation flag '-v 5', where I do not know how it performs cross-validation. I've never been skeptical about this before because I think there is an exact definition of cross validation. For 5-fold, divide by 5, train with 4/5, test with 1/5 and repeat this 5 times and average. $\endgroup$– Yalçın CenikCommented Mar 24, 2023 at 10:34
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I think you have two possibilities :
- Sampling error : that means your samples doesn't represent the population very well, in this case you'll need to add more samples in the zones that shows your model very weak, or maybe you could delete some bad samples to make the distribution of the reference response more normal
- Overfitting : that means you'll need a model regularisation (you have focused on the cost 'l2 penalty' but maybe it's not enough), google it until findin a regularisation algorithm that you could apply on your case
I think you should try the first one, it's more guaranteed Good luck
