# R mlr - How does tuneThreshold work?

I would like to tune the threshold for the following classification task using tuneThreshold in conjunction with a learner parameter.

I first tried to tune the threshold during the tuning of the learner by setting makeTuneControlRandom(..., tune.threshold = TRUE):

library(ElemStatLearn)
library(mlr)
data(spam)

lrn1 = makeLearner("classif.gbm", predict.type = "prob")
ps = makeParamSet(
makeIntegerParam("interaction.depth", lower = 1, upper = 5)
)
ctrl = makeTuneControlRandom(maxit = 2, tune.threshold = TRUE)
lrn2 = makeTuneWrapper(lrn1, par.set = ps, control = ctrl, resampling = cv2)
r = resample(lrn2, task, cv3, extract = getTuneResult)
print(r$extract) [[1]] Tune result: Op. pars: interaction.depth=4 Threshold: 0.52 mmce.test.mean=0.0586857 [[2]] Tune result: Op. pars: interaction.depth=5 Threshold: 0.54 mmce.test.mean=0.0557573 [[3]] Tune result: Op. pars: interaction.depth=5 Threshold: 0.51 mmce.test.mean=0.0514993  Here the optimal threshold is 0.51. I then tried tuning the threshold by using tuneThreshold directly on the prediction object: tuneThreshold(r$$pred)$$th [1] 0.5650756$perf
mmce
0.05303195


Here the optimal threshold is 0.565. I don't understand why the optimal threshold here is different from the one above, why didn't it return the same threshold as the one found above, i.e. 0.51? It seems to be adding another layer of randomness but I don't know where or how because when I call tuneThreshold(r$pred) again, the threshold and performance score do not change. How does tuneThreshold work exactly? What does it do with the prediction object r$pred?

tuneThreshold() simply determines the optimal threshold for predicting one class vs another given a set of predictions. That is, the only thing that is changed is the probability threshold that separates the classes; the predictions of the probability for each example are the same.
• Thank you very much for replying so quickly Lars. I understand that each learner configuration and CV fold may yield a different threshold and that resampling is only used to provide an unbiased estimate of the strategy's performance (so that if I'm happy with the performance, I then use train(learner, task) to train the learner on the full dataset) but what does tuneThreshold do when applied directly to r$pred directly via tuneThreshold? I've seen it being used in this way but it doesn't make sense to me. Also, where does this r[["pred"]][["threshold"]] come from? – user51462 Jan 26 at 0:24 • The predictions are the probabilities. The tuned threshold is the probability threshold for one class vs the other. Does that make sense? – Lars Kotthoff Jan 26 at 19:35 • Yes. My confusion is as to why tuneThreshold(r$pred) returns a single threshold when the CV results in 3 models (each with their own set of predictions)? Also, as a separate question, when I inspect the resampling object, I see an element of class ResamplingPrediction. In that element there is a slot for a single threshold (see r[["pred"]][["threshold"]]). This threshold is different from what tuneThreshold(r\$pred) returns and I was wondering how it was calculated? – user51462 Jan 26 at 22:10
• The result for tuneThreshold is across all predictions. How the threshold in the resampling prediction is computed depends on what you're doing with it. – Lars Kotthoff Jan 26 at 22:27