Best Practice for choosing hyperparameters for stacking/majority voting My goal is to get the best hyperparameters for the classifiers I used in Stacking/Majority Voting.
This is the steps that I did (with very small Titanic dataset):


*

*I use GridSearchCV to tune each classifiers (logreg, Random Forest, XGB, etc)

*I use the best performing classifier, with the best hyperparameters from step 1. I always use logreg as the 2nd classifier when Stacking


My questions are :


*

*Is it considered "best practice" to use the best hyperparameter of each classifier for Stacking/Majority Voting?

*I found out that its possible to do GridSearchCV  when Stacking (with mlxtend), so the chosen hyperparameters is the best for Stacking, not the best for each classifier (as opposed to the 1st point). However, is it worthed the effort? I believed the computational cost will be huge because there will be a lot of hyperparameters combination for GridSearchCV to try.

*Or do I missing something important here? Please kindly give me your practical advice


My current Stacking yields the best result, but only differ +- 1% from standalone advanced classifier (Random Forest/XGBoost)
 A: To give you a more specific answer:

could you explain more about "   Besides, you cannot expect better
  results whatever way of doing parameter search you prefer if your
  standalone classifiers produce 'intercorrelated' results"?

The best results are obtained using independent predictions of appriximately equal quality, the ones which are orthogonal. An opposite case is using predictors which 100% correlate with other models' predictions. Then you get no additional information at all: all predictions tell you the same thing.
Imagine you apply a not perfect ruler to get estimates of your desired quality. 


*

*Each measurement is independent of each other.  

*The ruler application produces a random error each time.


Here assumptions allow the Law of Large Numbers (LLN) to come into effect. If you average many independent noisy measurements many times (limit is infinity) you will obtain a measurement infinitely close to the real value (on population).
Consider now a situation with correlated outputs from your models (one model application to one observation is one application of a ruler). I mean your models produce correlated outputs per observation in your sample with regard to other models' predictions for the same observations.
Since they are not independent, but indeed correlated, the LLN does not correctly apply here since the main assumption is broken. Hence you cannot expect to get an unbiased estimate even if you try lots of different models.

And I don't get why tuning multiple model hyperparameters at the same
  time will lead to overfiting?

Suppose you have 100 reasonable combinations of parameters per model (like, regularization coefficient, interaction depth, boosting iterations). If you feat 3 models separately, you parse through 300 possibilities. If instead of evaluating each model separately you decide to optimize the parameters for all models at once, you get 100 to the power of n model number. It can be 1 000 000 different model combinations if you have just 3 models... You really just create more hypotheses to check, each hypothesis is whether my model superposition is a good estimator of the modeled data.
The more the number of models to choose from the higher the chances that on train and validation you will get at least 1 model superposition which will show good estimates everywhere but on test.
