I have some data which I use SVC models with 10 fold cross validation and a parameter grid search on (scikit.learn). I observed that the predictions of some folds have low accuracy, whereas remained folds have better accuracy. With different C and gamma parameters of SVC, the prediction of previously low folds improves, but the better accuracies of other folds become worse than before. From this I derive, that if I could somehow combine these two or three models with different parameters, it will overall get better results. As far as know, Python has ensemble classifiers, but it allows only one base model, but in my case there will be more than one SVC models which should have different parameters.

My question is: is it possible to combine different SVC models with different parameters in Python and/or scikit.learn? If so, how can I do it?


First: yes, your assumptions that combining such models might be beneficial are reasonable (will likely reduce the overall variance). Consider also combining different models types if this is possible in your setup.

There would be different options to do so, like:

  • Model averaging
  • Bagging
  • Boosting

You could easily implement model averaging yourself, independently of any framework you use for training your models (regression: average the prediction of a new sample from all models you obtained during training; classification: derive class probabilities from the amount of classifications for each class instead).

Bagging is essentially the same, but uses subsets of training data (=not all samples) to train each model, which makes all the models different. Again, you could easily do this yourself. After training the models, the prediction process is the same as for model averaging.

Only for boosting you probably need to use a framework instead, as it is a bit more complicated - so there you would need to look into the details of the scikit.learn API if it allows for combining different models with different hyperparameters.

One more thing: be aware that by doing something like this after model evaluation, you will likely still overfit your problem. You should think about incorporating the parametrization of your ensemble (nr. of models etc.) into your evaluation routine as well.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.