I am performing a k fold cross validation in order to fit the cost parameter of a linear SVM model (I am using the LiblineaR package in R). So for each cost value I have k models, each trained on a different but overlapping sample of the dataset. The next step is then to train a model on the entire training set and test on a validation set not included in any of the k folds - standard.

For the problem I am tackling, false positives are considered more costly than false negatives, therefore on each of the k folds I am tuning the decision threshold based on a weighted accuracy measure. My question is, once the decision threshold has been chosen for each of the k models, does it make sense take the average of these decision thresholds, and apply this to the final model output?

For example suppose a cost of 1 turns out to be the best, and I have k=4 models trained with a cost of 1. The tuned decision thresholds for these 4 models are 0.12, 0.14, 0.04, 0.02 and the average is 0.08. If I then train a model using a cost of 1 on the entire dataset, can I use 0.08 as the decision threshold?

Or are the decision values from models trained on different samples not comparable, in which case taking the mean would be meaningless.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.