1
$\begingroup$

I am performing a k fold cross validation in order to fit the cost parameter of a linear SVM model (I am using the LiblineaR package in R). So for each cost value I have k models, each trained on a different but overlapping sample of the dataset. The next step is then to train a model on the entire training set and test on a validation set not included in any of the k folds - standard.

For the problem I am tackling, false positives are considered more costly than false negatives, therefore on each of the k folds I am tuning the decision threshold based on a weighted accuracy measure. My question is, once the decision threshold has been chosen for each of the k models, does it make sense take the average of these decision thresholds, and apply this to the final model output?

For example suppose a cost of 1 turns out to be the best, and I have k=4 models trained with a cost of 1. The tuned decision thresholds for these 4 models are 0.12, 0.14, 0.04, 0.02 and the average is 0.08. If I then train a model using a cost of 1 on the entire dataset, can I use 0.08 as the decision threshold?

Or are the decision values from models trained on different samples not comparable, in which case taking the mean would be meaningless.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.