First, I tried not tuning the hyperparameters without setting tune.threshold=TRUE.
lrner1 = makeLearner(learner, predict.type = "prob" )
ctrl = makeTuneControlRandom(maxit = 10) # , tune.threshold=TRUE
lrn2 = makeTuneWrapper(lrner1, par.set = num_ps, control=ctrl, resampling = rdesc, measures=list(mlr::f1, tpr, tnr, setAggregation(f1, test.sd)))
r = resample(lrn2, task_dataset, cv3, extract = getTuneResult)
> r$extract
[[1]]
Tune result:
Op. pars: cost=3; nu=-4; degree=10
f1.test.mean=1.0000000,tpr.test.mean=1.0000000,tnr.test.mean=1.0000000,f1.test.sd=0.0000000
[[2]]
Tune result:
Op. pars: cost=5; nu=3; degree=8
f1.test.mean=1.0000000,tpr.test.mean=1.0000000,tnr.test.mean=1.0000000,f1.test.sd=0.0000000
[[3]]
Tune result:
Op. pars: cost=5; nu=1; degree=5
f1.test.mean=1.0000000,tpr.test.mean=1.0000000,tnr.test.mean=1.0000000,f1.test.sd=0.0000000
The f1 statistic (and other accuracy metrics) is very high at the default threshold of 0.5. However, when I specify that tune.threshold=TRUE, all of the iterations have much lower accuracy stats.
lrner1 = makeLearner(learner, predict.type = "prob" )
ctrl = makeTuneControlRandom(maxit = 10, tune.threshold = TRUE)
lrn2 = makeTuneWrapper(lrner1, par.set = num_ps, control=ctrl, resampling = rdesc, measures=list(mlr::f1, tpr, tnr, setAggregation(f1, test.sd)))
r = resample(lrn2, task_dataset, cv3, extract = getTuneResult)
> r$extract
[[1]]
Tune result:
Op. pars: cost=1; nu=0; degree=3
Threshold: 0.98
f1.test.mean=0.2133333,tpr.test.mean=0.1500000,tnr.test.mean=1.0000000,f1.test.sd=0.3069564
[[2]]
Tune result:
Op. pars: cost=3; nu=5; degree=9
Threshold: 1.00
f1.test.mean=0.0000000,tpr.test.mean=0.0000000,tnr.test.mean=1.0000000,f1.test.sd=0.0000000
[[3]]
Tune result:
Op. pars: cost=5; nu=2; degree=3
Threshold: 0.99
f1.test.mean=0.1466667,tpr.test.mean=0.0900000,tnr.test.mean=1.0000000,f1.test.sd=0.2022100
Also, every time I do it, the thresholds are very high.
Is something wrong is or this behavior expected?
