Questions tagged [stacking]

Stacking is a meta-ensemble machine learning technique that trains a second-level machine learning model on the predictions from multiple machine learning models trained on the data.

Filter by
Sorted by
Tagged with
3
votes
0answers
20 views

AIC model averaging when models are correlated

AIC model-averaging: In "standard" AIC model averaging we average models with weights proportional to $$w_i \propto \exp( -0.5 \times \Delta \text{AIC}_i ),$$ where $\Delta \text{AIC}_i$ is ...
0
votes
0answers
8 views

Basic Questions on Stacking (ensemble models)

I found a paper online "Popular Ensemble Methods: An Empirical Study" (Opitz, Maclin, 1999). Was this really the first observed use of "model stacking" (ensemble learning) in ...
1
vote
0answers
52 views

Model stacking with windowed features - info leakage?

This is similar to other questions on leakage (for example, this post), but all of my data are generated with look-back features, and nothing can be assumed to be iid. I'm curious if you think there ...
0
votes
0answers
25 views

StackingClassifier + RandomSearchCV: How is it dividing the folds under the hood?

I'm able to (based on the example from the accepted answer here ) set up a StackedClassifier and add RandomSearchCV to perform a quick hyperparameter search. The models/pipelines are set up like in ...
1
vote
2answers
30 views

How to combine outputs from different non-machine learning prediction models?

I have a list of cancer mutations and I have run five different bioinformatics tools that predict whether a particular mutation will lead to a more aggressive form of cancer or not (0-neutral;1-...
1
vote
0answers
23 views

Should we train base learners on same folds when we stack different models

I am confused about k-fold stacking. Should we train all the base learners on same folds? I mean is it ok to do the KFold split with different seeds or all base learners should trained on folds ...
0
votes
0answers
12 views

Aggregation model in stacking with or without initial features

I am implementing a Stacking Ensemble, which works in general, for Supervised problems. Ideally, this was my idea: Train: I take the training-set and after training the N models of the ensemble, I ...
2
votes
1answer
40 views

Model stacking, Super Learner Algorithm

I've recently started studying ensembles in ML, particularly Super Learner Algorithm. To be honest, although I have read several articles related to this topic, I am a little bit confused. I want to ...
4
votes
1answer
96 views

Cross Validation in StackingClassifier Scikit-Learn

In Scikit-Learn StackingClassifier documentation it's written: Note that estimators_ are fitted on the full ...
0
votes
0answers
26 views

Stacking and Ensembling methods in Data Science

Recently it seems that stacking and ensembling methods have become more popular, and using these methods can give better results than using a single algorithm. My question is: What are the reasons, ...
0
votes
0answers
19 views

Combining two regression models using a meta-classifier

I know in stacking it is possible to build meta-regression models ontop of regression models to improve performance (meta-regression models for regression models) but are there any techniques if ...
0
votes
0answers
11 views

repeat time point linear regression in R

I have this data set and want to run mixed linear model. I am trying to study the effect of the binary variable "medicine" on the outcome (which is repeated measures of lab values at day 1, 2 and 3 - ...
0
votes
0answers
16 views

Stacking model: is finding best threshold necessary?

After performing a stacking model (with rstudio), is it necessary to choose the best threshold for it? In general after finding the best model among all the fitted models , you have to choose the ...
0
votes
1answer
41 views

Why stacking use strong learner as base learner?

I am wondering why stacking uses strong learners as base learners. How to understand it from expectation and variance way, or bias and variance way?
1
vote
0answers
16 views

Building a set of time-dependent ensemble models: Can I stack different models at each time step?

I am trying to predict the values of a physical quantity $A(t)$ at different time steps $t$ using Machine learning. The thing is that, because my training data are time dependent, I have to make a ...
0
votes
0answers
58 views

Stacking using layer-1 models predictions on test set

I am new to Data Science and have been studying the methods of stacking to find out if it can meet the following fact, but I did not find or understand evidence that it can or cannot work. Let's ...
1
vote
1answer
158 views

Is this kind of stacked ensemble method prone to over-fitting?

I am working on a stacked ensemble method. I trained three classifiers as my first-layer models and one Logistic Regression as my second layer model. I then stacked both the first-layer models and ...
3
votes
1answer
89 views

About the need of splitting data in stacking

I learned stacking of machine learning in a book, hands-on machine learning 2nd edition (2019). The picture was cited from hands-on machine learning 2nd edition (2019). In the above situation, ...
1
vote
0answers
175 views

How to create a test set in stacking when doing cross validation

I am using Weka to implement stacking with k-fold cross validation. As I understand, we first divide our dataset in to k folds, then we use k-1 folds for training and 1 fold for testing. This ...
1
vote
0answers
17 views

Should initial features be added as input of meta-learner in stacking?

I'd like to know your opinion and reference on adding initial features (i.e. the ones used to train the weak learners) to the input of the meta learner (aside of the predictions from weak learners) ...
2
votes
0answers
17 views

How to optimize hyperparameters in stacked model?

I was wondering whether somebody could explain how to optimize hyperparameters for the base learners and meta algorithm when stacking? In many tutorials they seem to be plucked out of thin air! ...
0
votes
2answers
533 views

Why is my stacked model worse than my base models? [closed]

I'm learning stacking and start with the approach outlined in Introduction stacking I've plotted the data: I first would like to check if my algorithm is correct (see below): So I basically ...
0
votes
1answer
48 views

How valid is this Stacking Model (input features to weak learners are different)?

I have a set of features with 6 of them being categorical, 1 continuous and 2 textual in type. I have to predict the labels ( 10 in number) for them. I tried applying several models and came to a ...
0
votes
0answers
24 views

Are there specific Machine Learning Algorithms that are more indicated for Real Time Analytics?

As the title suggests, I am wondering if there are specific ML algorithms that are more suitable for real time learning. In my case, I am working on deploying a stacking algorithm on Spark Streaming ...
1
vote
2answers
59 views

I am learning to do stacking. I want to know what will the input be to level 2 classifiers [closed]

In classification/regression problems, say if we use five different base classifiers, we get 5 predictions for each example. What would be the input to the second level classifiers?
57
votes
7answers
7k views

Industry vs Kaggle challenges. Is collecting more observations and having access to more variables more important than fancy modelling?

I'd hope the title is self explanatory. In Kaggle, most winners use stacking with sometimes hundreds of base models, to squeeze a few extra % of MSE, accuracy... In general, in your experience, how ...
1
vote
0answers
50 views

Decrease the variance of ExtraTrees Classifier

I am trying to solve a machine learning problem. I am using ExtraTrees Classifier. When I am plotting the learning curves, I can see a wide gap. I need to decrease that (variance). I read about ...
1
vote
0answers
294 views

Why using Out-of-fold predictions as metafeatures in stacking?

So my question is essentially the same as this one: Why do we generate out-of-fold predictions for meta-ensembling/stacking? However, I am not entirely satisfied with the answer (not detailed enough ...
4
votes
2answers
2k views

Stacking without splitting data

I learned Stacking used in Ensemble learning. In Stacking, training data is split into two sets. The first set is used for training each model (layer-1, left figure), the second one is used for ...
0
votes
1answer
424 views

Generate ensemble of classifiers based on predefined feature subsets in R using mlr

I would like to create an ensemble classifier for a dataset and use different classification models for different subsets of features (these feature subsets are predefined as the data set I am working ...
1
vote
2answers
2k views

Why do we generate out-of-fold predictions for meta-ensembling/stacking?

Here's the guide I'm looking at: http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/ Here's the relevant excerpt: The main point to take home is that we’re using the ...
3
votes
1answer
837 views

Combining bagging and stacking, with and without clusters and heteroskedasticity

Question 1: Start with the classing case of bagging, say in random forest. Fit $B$ trees to bootstrap samples of the data. Average the predictions of the $B$ trees to form a final prediction. ...
1
vote
1answer
181 views

Stacking or Voting - Multiple feature sets extracted with different parameters

I am extracting features from time series data using different parameters and then creating a SINGLE feature based data set with all features to perform classification. If I wanted to create separate ...
2
votes
1answer
978 views

Model stacking, what is the input of meta classifier?

I know that by stacking different models among which there has a low correlation can boost the performance of on single model. And I found a picture In step 7, the $h_j(x_i)$ in new data $x_i^{'}=\{...
1
vote
0answers
126 views

Stacking Lasso models

let say I have M subsets of independent variables and I want to use stack learner to predict dependent variable y. for each subset I use lasso method to get meta features (predictions). I have 2 ...
0
votes
0answers
409 views

Stacking, How can I check the correlation of models?

In ensemble learning, model stacking is a good way to improve the performance of a single model. However, models chosen to be stacked must have least correlation in order to exploit the performance of ...
4
votes
1answer
283 views

Can we stack the strong learners? [duplicate]

Stacking is done with combining all the weak learners. What will happen if we do it with strong learners? A case of overfitting?
2
votes
3answers
231 views

Models combinations

My goal is time series forecasting. I have created a number of models to make predictions. I know that forecast quality can be improved by combining predictions from different models(linear ...
6
votes
1answer
8k views

How to stack machine learning models in R

I am new to machine learning and R. I know that there is an R package called caretEnsemble, which could conveniently stack the models in R. However, this package looks has some problems when deals ...
1
vote
2answers
361 views

Using one ml models output to choose another models input

I'm dealing with a low event rate problem (e.g. credit card fraud). I've balanced my data with SMOTE, and ran a neural net model (cross validated with recall as the measure). However my precision (...
1
vote
0answers
34 views

Using different data transformation in ensembles

I have trained few models using sklearn and python. However I have scaled the data for Support Vector Machines - standarized and for Neural Network - Scaled [0-1] since it gives me better results and ...
2
votes
0answers
452 views

Why StackingRegressor doesn't catch the trend?

I just reviewed very good example of fitting StackingRegressor from mlxtend package. ...
5
votes
2answers
3k views

Model Stacking - Gives poor performance

I'm trying model stacking in a kaggle competition. However, what the competition is trying to do is irrelevant. I think my approach of doing model stacking is not correct. I have 4 different models: ...
1
vote
2answers
201 views

how do you stack models with 100% training accuracy?

Suppose I have several models, one of them $M$ has a 100% training accuracy. So regardless of how a stack the models, the stacked model is just M. e.g. If I use a linear model to stack them, then the ...
1
vote
1answer
578 views

Ensemble Decision Trees and Gradient Boosted Decision Trees

I see people often ensemble Gradient Boosted Decision Trees and Random Forests together. Does it make sense to ensemble a Decision Tree and a GBDT together? Isn't this DT already a part of GBDT?
1
vote
0answers
86 views

Stacking sensitivity analysis

I conducted stacking of three algorithms (NN, J48 and BN) with logistic regression as the meta-classifier. I am interested in doing a sensitivity analysis so I am able to rank the predictors and ...
1
vote
1answer
176 views

Training separate models for sets of features?

I'm wondering if it is sensible to train different models for different sets of features (I do not mean one model for each feature). Say I have 4 features - 2 are nominal while the other two are 2 ...
-1
votes
1answer
655 views

Stacking Ensemble Meta Learning Components

I have been learning about Ensemble Algorithms. Reading this paper, I want to know about the meta classifiers that work well with Stacking Ensemble Learning Algo, since this work just talks about ...
0
votes
1answer
58 views

How do you calculate the covariance “stack up” of relative measurements?

Say that you have a measurement of $x_1$ relative to some globally fixed datum and a covariance for it, $\sigma_1$. If you have another measurement, $x_2$, taken relative to $x_1$ with its own ...
1
vote
1answer
184 views

Ensemble of boosting models

Are adaboost and gradient boost models highly correlated? Will including both the above models in ensemble improve accuracy significantly?? Because if the models are uncorrelated in the ensemble they ...