0
$\begingroup$

I am using an ensemble of rf,knn,xgboost and logistic regression to predict final probabilities. I am taking weighted average of all the prob predicted by individual models. I would like to know wouldn't this cause overfitting and I wont be able to generalize this for some other test set? As I am taking average of prob for one single test set and assigning weights for which the AUC on that particular test set is the most. Wont this cause the model prob to learn only one single test set? What else should I do to ensemble if it does indeed causes overfitting.

$\endgroup$
2
$\begingroup$

You should not be setting weights for which the AUC is the maximum for the particular test set. The test data should not be used for training the model at all.

This will give you the difference between the different types of datasets

You can use cross validation to prevent over-fitting, the basics of which can be found here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.