Bagging models with different metaparameters versus cross validation? If we have a model with a metaparameter C, the usual way to tune this parameter is via cross-validation (or building a validation set). The generalization error of the model with the optimal is evaluated on a test set. This is the approach I have always seen.
Another approach could be to choose C from a distribution of values and than aggregate the results in some way, e.g. via average or stacking.
Is there a reasoning to prefer the first or second approach ? Is the second approach used commonly ?
 A: From a certain point of view, optimizing the meta- or hyperparameter is the opposite approach to ensemble models. They are good for sifferent situations.
The point of ensemble models (stacked or bagged) is to reduce variance in the submodels due to the imperfect training (e.g. too few training cases). Bagging does not change the prediction and thus does not help when the submodels yield the same (stable) predictions (for the same case). The argument for stacking works in the same way.
In contrast, when optimizing model meta-/hyperparameters, this will almost always include an optimization of the model complexity. If the results are not stable, we'd usually say the optimization procedure was not (fully) successful.
Advanced model opimization techniques include a bias towards less complex models, i.e. their target functional is not only observed performance but includes a penalty for instability (e.g. the one-standard-deviation heuristic).
Last but not least, you may find a stable result set of hyperparameters in the optimization, but still get models whose prediction is unstable. In that case, you may want to combine both approaches and build a bagged predictor with the optimized hyperparameter set.
