I am using an ensemble of rf,knn,xgboost and logistic regression to predict final probabilities. I am taking weighted average of all the prob predicted by individual models. I would like to know wouldn't this cause overfitting and I wont be able to generalize this for some other test set? As I am taking average of prob for one single test set and assigning weights for which the AUC on that particular test set is the most. Wont this cause the model prob to learn only one single test set? What else should I do to ensemble if it does indeed causes overfitting.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
You should not be setting weights for which the AUC is the maximum for the particular test set. The test data should not be used for training the model at all.
This will give you the difference between the different types of datasets
You can use cross validation to prevent over-fitting, the basics of which can be found here
