R mlr - How does tuneThreshold work? I would like to tune the threshold for the following classification task using tuneThreshold in conjunction with a learner parameter.
I first tried to tune the threshold during the tuning of the learner by setting makeTuneControlRandom(..., tune.threshold = TRUE):
library(ElemStatLearn)
library(mlr)
data(spam)
load_all()

task = makeClassifTask(data = spam, target = "spam")
lrn1 = makeLearner("classif.gbm", predict.type = "prob")
ps = makeParamSet(
  makeIntegerParam("interaction.depth", lower = 1, upper = 5)
)
ctrl = makeTuneControlRandom(maxit = 2, tune.threshold = TRUE)
lrn2 = makeTuneWrapper(lrn1, par.set = ps, control = ctrl, resampling = cv2)
r = resample(lrn2, task, cv3, extract = getTuneResult)
print(r$extract)

[[1]]
Tune result:
Op. pars: interaction.depth=4
Threshold: 0.52
mmce.test.mean=0.0586857

[[2]]
Tune result:
Op. pars: interaction.depth=5
Threshold: 0.54
mmce.test.mean=0.0557573

[[3]]
Tune result:
Op. pars: interaction.depth=5
Threshold: 0.51
mmce.test.mean=0.0514993

Here the optimal threshold is 0.51.
I then tried tuning the threshold by using tuneThreshold directly on the prediction object:
tuneThreshold(r$pred)
$th
[1] 0.5650756

$perf
      mmce 
0.05303195 

Here the optimal threshold is 0.565. I don't understand why the optimal threshold here is different from the one above, why didn't it return the same threshold as the one found above, i.e. 0.51? It seems to be adding another layer of randomness but I don't know where or how because when I call tuneThreshold(r$pred) again, the threshold and performance score do not change. How does tuneThreshold work exactly? What does it do with the prediction object r$pred?
 A: tuneThreshold() simply determines the optimal threshold for predicting one class vs another given a set of predictions. That is, the only thing that is changed is the probability threshold that separates the classes; the predictions of the probability for each example are the same.
In your first example, you're also tuning a hyperparameter of the learner. This will result in different predictions for each given data point, depending on the setting of that hyperparameter. The threshold is then tuned as outlined above for a set of given predictions. There's no single threshold that is best for all hyperparameter settings and the predictions they produce, so you're getting different thresholds for different hyperparameter values (in your output you actually have three different interaction depth-threshold combinations, one for each fold).
Now all of this also depends on the data. As you can see in your code, you're ending up with the same hyperparameter setting, but different thresholds for different folds in your CV -- that's because the data points used in those evaluations are different and hence yield different results.
So how do you determine the optimal values to use? You probably want to use the full data set to determine that (e.g. for the threshold use the predictions on the entire data), using a model that you determined to have good performance with a CV -- the point of resampling in general is to give you an estimate of how good the particular approach would perform on new data to enable you to choose the best approach.
