is it pointless to stack models of different data types/structure?

Question

I am studying ensemble methods machine learning, in particular I am focusing on stacking.

In stacking different models are used to get an output. Then all the outputs are 'combined' together to build a model. So, first of all, I guess the models MUST be different between each other (does it make sense?). Once we have these models, does it make sense to apply them to different data types/structures?

UPDATE: For “data structures” I am just saying different methods that learn different models (e.g. a degree 2 polynomial, a degree 4 polynomial, a KNN classifier, a SVM, etc.).

Answer 1

Using Wikipedia's definition

In computer science, a data structure is a data organization, management, and storage format that enables efficient access and modification. More precisely, a data structure is a collection of data values, the relationships among them, and the functions or operations that can be applied to the data, i.e., it is an algebraic structure about data.

Calling things like "a degree 2 polynomial, a degree 4 polynomial, a KNN classifier, a SVM, etc." is unfortunate. Those are machine learning algorithms, they have nothing to do with data structures.

Machine learning model is a mathematical representation of the data, partially assumed (structure), partially learned from the data (parameters). Things such as linear regression, polynomial regression, $k$-NN, random forest, would all be refered as models. For learning the model we have machine learning algorithms. People would often use the terms exchangably.

By stacking models we mean stacking any kind of models. You could stack dofferent linear regression models, that differ by the data that was used for training them, and hence by the parameters, but also you can stack together $k$-NN, linear regression, SVM, and random forest. Usually, stacking diverse models gives better results than for homogeneus models, so it would be perfectly reasonable to stack together $k$-NN, SVM, and polynomial regression.

Answer 2

First, let us review what stacking is. Then we can answer the question. Stacking is a machine learning variation of meta-analysis. I don't claim any originality for this answer from the geeksforgeeks site, which says,

"There are many ways to ensemble models, the widely known models are Bagging or Boosting. Bagging allows multiple similar models with high variance are averaged to decrease variance. Boosting builds multiple incremental models to decrease the bias, while keeping variance small.

Stacking (sometimes called Stacked Generalization) is a different paradigm. The point of stacking is to explore a space of different models for the same problem. The idea is that you can attack a learning problem with different types of models which are capable to learn some part of the problem, but not the whole space of the problem. So, you can build multiple different learners and you use them to build an intermediate prediction, one prediction for each learned model. Then you add a new model which learns from the intermediate predictions the same target. This final model is said to be stacked on the top of the others, hence the name. Thus, you might improve your overall performance, and often you end up with a model which is better than any individual intermediate model. Notice however, that it does not give you any guarantee, as is often the case with any machine learning technique."

Note my bolding of text above; no guarantees it works. Moreover, they outline a procedure for stacking:

"How stacking works?

1. We split the training data into K-folds just like K-fold cross-validation.
2. A base model is fitted on the K-1 parts and predictions are made for Kth part.
3. We do for each part of the training data.
4. The base model is then fitted on the whole train data set to calculate its performance on the test set.
5. We repeat the last 3 steps for other base models.
6. Predictions from the train set are used as features for the second level model.
7. Second level model is used to make a prediction on the test set."

Also, see this other article same website.

Now the answer Stacking uses different models for the same data. To combine results from different data sets would be a more general meta-analysis, and there appears to have been some work on that topic. The OP asks Once we have these models, does it make sense to apply them to different data types/structures? It makes sense to go through the same procedure from scratch. That is, if the problem changes, the models needed will be different. If the problem doesn't change and there is merely more of the same type of data, then the analysis can be repeated on that larger data set.