Questions tagged [ensemble-learning]

In machine learning, ensemble methods combine multiple algorithms to make a prediction. Bagging, boosting and stacking are some examples.

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

Combining regression models based on missing data patterns

I have a dataset that contains a few patterns of missingness. For this dataset, I have a training set that is complete and contains all input features. My test set has complete observations for the ...
Squan Schmaan's user avatar
2 votes
1 answer
34 views

Ensemble Methods for Probabilities

I am currently trying to build a stacked algorithm in order to determine how many people in each region of a country will be likely to buy a product versus its competitors. I have some data from an ...
huntercallum's user avatar
0 votes
0 answers
26 views

Ensemble Random Forest Overfitting

I am running an ensemble random forest model (a newer method published in 2020). The model works by using a double bootstrapping step to balance imbalanced training data. Then you grow multiple ...
Greatwhite4's user avatar
0 votes
1 answer
53 views

Bagging Ensemble Math

You are working on a binary classification problem with 3 input features and have chosen to apply a bagging algorithm (Algorithm X) on this data. You have set max_features = 2 and n_estimators = 3. ...
Tanjim Taharat Aurpa's user avatar
1 vote
0 answers
18 views

Cross validation + model stacking with hyperparameter tuning while sharing data?

Let's say we want to stack 2 base models: an XGBoost regressor and a deep neural network by linearly combining their predictions as ...
statnoob's user avatar
1 vote
0 answers
20 views

Is there a known way of producing forecasts with reasonable fit and residuals that are at least independent, & ideally negatively correlated? [closed]

I am trying to do some forecasts. I have produced multiple forecasts by a variety of methods. All of the forecasts I have generated so far have residuals that are strongly positively correlated.I ...
andrewH's user avatar
  • 3,117
0 votes
0 answers
6 views

How should I test whether simultaneous residuals from two models of the same time series are independent?

Suppose I have two different models, with comparable goodness of fit but very different structure, that I have fit to the same time series. For both, the residuals pass various tests of normality. How ...
andrewH's user avatar
  • 3,117
0 votes
0 answers
25 views

Should I create an ensemble by averaging deep models' weights and biases?

When I train deep models with cosine annealing learning-rate scheduling and warm restarts, I get models that achieve completely different scores on my validation set, after each training cycle. There ...
SomethingSomething's user avatar
1 vote
0 answers
32 views

Estimation under model uncertainty that cannot be adjudicated empirically

Many well-known methods address specific forms of model uncertainty that can be adjudicated empirically. For example, if we are fitting a predictive model and there is uncertainty about the set of ...
half-pass's user avatar
  • 3,740
4 votes
0 answers
86 views

Why are approaches that approximate a random forest with a single decision not more popular?

I understand that random forests yield better performance than standard decision trees, but are less interpretable, because they do not generate a single tree. In this question, several users provided ...
nka5we's user avatar
  • 49
1 vote
0 answers
50 views

Weighted bootstrap sampling vs. uniform bootstrap sampling with later weighting

Assume I have a fancy procedure $w: X \to \mathbb{R}$ to come up with weights for examples $x \in X$. Think of it as similar to the weights used in e.g. some boosting procedures. Now, I want to build ...
ngmir's user avatar
  • 341
2 votes
1 answer
56 views

Should I apply normalization to predicted probabilities from 7 different models before computing correlation among them?

I'd like to check if there are correlation among predicted probabilities of models in a voting classifier. According to the table below, one of models, Model5, has mean 40.9% and standard deviation 46....
KAI's user avatar
  • 23
2 votes
1 answer
90 views

Are Bagged Ensembles of Neural Networks Actually Helpful?

I've been looking into ways to estimate uncertainty for regression tasks on neural networks. One of the obvious options is ensemble modeling. Consider an ensemble of neural networks that all have ...
noNameTed's user avatar
  • 113
2 votes
1 answer
36 views

Ensemble learning with models of different quality. Develop a voting method that takes accuracy, F1, recall, calibration of each model into account

Lets assume I have 24 random forest models. Each of 24 random forest models produces a class prediction. I am currently using simple majority voting to select final prediction. Can someone please ...
MSKO's user avatar
  • 61
3 votes
1 answer
271 views

Fitting a simple model first, then training a neural network on the error

Can someone tell me what the name is for the following process? I have some data with inputs $x_i$ and outputs $y_i$, and I fit a simple model (e.g. linear regression) to them. Then, I compute the ...
Fai Wang's user avatar
1 vote
1 answer
361 views

Forecasting a Time Series Model for 1000s of Time Series

I'm currently immersed in a challenging forecasting project centred around predicting the required work hours to complete various tasks within a team setting. My dataset comprises crucial attributes, ...
Tirth's user avatar
  • 13
1 vote
0 answers
16 views

Least squares with multiple outputs but one coefficient per example

According to Elements of Statistical Learning Ch 8.8, we can apply least sqaures at the population level to show that for a regression ensemble $f_1(x), f_2(x), \ldots , f_M(x)$ where $f_j: \mathbb{R}^...
Seraf Fej's user avatar
  • 546
2 votes
0 answers
98 views

Why would a model combining two pre-trained models not even achieve the performance of the best sub-model?

I have two different CNNs trained on the same dataset. One performs a bit better than the other but I believe each can provide different and useful information. I use ...
raquelhortab's user avatar
0 votes
0 answers
43 views

Training a variable-length document-level LSTM on variable-length sentence-level data using ngrams and majority voting

Let's assume that I want to train a classification model for document-level input. The input is sequential (a sequence of tokens within a text). Documents may vary in length (i.e., one or multiple ...
Damiaan Reijnaers's user avatar
1 vote
1 answer
56 views

Why should a valid diversity measure be independent of the target variable?

Consider an ensemble of weak learners (i.e. regressors or classifiers) whose predictions are aggregated (e.g. via averaging or majority vote) into an ensemble estimate. This gives rise to the question ...
ngmir's user avatar
  • 341
2 votes
0 answers
42 views

Is bagging less useful in 'big data' settings?

In 'big data' settings where the number of samples $n$ may be very large (for fixed number of features), is bagging less or more effective at reducing variance? I heard the claim that it is less ...
WeakLearner's user avatar
  • 1,481
0 votes
0 answers
32 views

How to choose the meta learner for the super learner model?

I am building a super learner ensemble model using the classifiers SVM, kNN, AdaBoost, XGBoost, and Random Forest. However I am not sure the logic behind what classifier to use for the meta learner I ...
Lucy's user avatar
  • 1
4 votes
1 answer
940 views

SHAP values of Ensemble Model

I predict a continuous variable by taking the average of $N$ model predictions. The models are different in terms of their functional form, i.e. a tree model, a neural net, etc. Is the average SHAP ...
shenflow's user avatar
  • 1,109
1 vote
0 answers
19 views

Approaching multiple records for one observation; radiomics of 2D slices of a 3D object

Background I am trying to create a model that can predict Type 2 diabetes in a patient based on MRI scans of their thigh muscle. Previous literature has shown that fat deposition in the muscle of ...
Saminy Creed's user avatar
1 vote
1 answer
109 views

Weights Update - Ensemble Models

I must identify if a data point is an outlier or not in a dataset (we don't have labels). I have different unsupervised models to identify the outlier. Then, I normalize the outlier score and I ...
AdrianJoi's user avatar
0 votes
0 answers
54 views

Is Endogeneity an assumption of Ensemble Methods?

I am using catboost regressor and lgbm regressor to perform regression on dataset. I want to know the assumptions of both the models. Where can I find assumptions for both the models? Next I want to ...
Lopez's user avatar
  • 147
0 votes
2 answers
248 views

Combining logistic regression and decision tree?

I'm working on a project classifying patients as having (1) or not having (0) a particular condition. Someone I work with has suggested fitting a decision tree on this data, and using the leaf node ...
k13's user avatar
  • 47
1 vote
2 answers
271 views

Do random forests use weak learners (like XGBoost) or fully grown trees?

So it sounds like boosting techniques (eg. XGBoost) uses weak learners (stumps) to gradually learn sequentially. This is not in dispute I hope. However, with bagging techniques (eg. Random Forest) I'm ...
Katsu's user avatar
  • 911
2 votes
1 answer
86 views

Why doesn't boosting assign higher weight to the "good" (low residual) models?

Extremely confused about the following: Lets say we start out with a dumb weak learner. Since its the 0th model and hasnt learned anything yet, we have a high residual, lets say of 10,000. We produce ...
Katsu's user avatar
  • 911
0 votes
1 answer
254 views

Subset Differences between Bagging, Random Forest, Boosting?

Per my understanding, there are 2 kinds of "subsets" that can be used when creating trees: 1) Subset of the dataset, 2) Subset of the features used per split. The concepts that I'm comparing ...
Katsu's user avatar
  • 911
1 vote
1 answer
228 views

Evaluating Feature Importance for a Super Learner Ensemble Meta-Model

I have been reading up on super learner ensemble methods that utilize multiple models and model configurations to make model predictions as good or better than any individual base model previously ...
Jake Niederer's user avatar
0 votes
0 answers
27 views

Do the neural networks belonging to a deep ensemble need to be trained on the same training set?

As the title says, I was wondering, if I have to train every neural network of a deep ensemble on a different training set or on the same one. I ask this question because I am getting weird results. ...
Alucard's user avatar
  • 325
0 votes
0 answers
45 views

Math behind ensemble learning

I'm struggling to find some clear math behind ensemble learning. I can simulate it very easily, eg: ...
Blaze's user avatar
  • 73
5 votes
1 answer
158 views

Boosting definition clarification

Regarding boosting in the context of machine learning. One definition I have encountered talks about turning multiple weak learners into one strong learner, and another talks about starting with a ...
A. Maman's user avatar
  • 212
1 vote
0 answers
15 views

Can someone explain why finding similar embeddings coming from two different net gives bad recall?

I'm currently working on an ensemble of 5 differently trained networks using MinkLoc3D v2 as base-net. I'm currently investigating the reason for lousy recall when I compare the extracted database ...
JackFrost67's user avatar
0 votes
0 answers
139 views

What happens to the accuracy of a decision tree after pruning?

What happens to the accuracy of a decision tree when it is pruned? Can be higher than the accuracy of the fully-grown decision tree?
san's user avatar
  • 11
0 votes
0 answers
87 views

Ensemble learning with different data sets

Consider model A, a deployed model that produces a probabilty of if an event occur or not for a population. This I want to improve by building another model, model B, on top of model A. Model B should ...
Henri's user avatar
  • 243
3 votes
1 answer
379 views

How to prove error of ensemble model by using the Hoeffding's inequality?

Under Binary classification situation, error between function $f$ and basic learner(classifier) $h_i(x)$ is $$P(h_i(x)≠f(x))=\mathcal{E}.$$ It is assumed that $T$ basic classifiers are combined by a ...
Kombatant82's user avatar
3 votes
0 answers
30 views

Is model selection itself a model?

Suppose that I wanted to choose from, for example, $Y = aX + \epsilon$ and $Y = aX^2 + \epsilon$. Is this meaningfully different from fitting $Y = a_1X + a_2X^2 + \epsilon$ and heavily penalizing $...
llllvvuu's user avatar
  • 223
0 votes
1 answer
32 views

How does accuracy increase in ensemble learning?

I have a doubt from a passage in the ensemble learning chapter of Aurelien Geron's book "hands on machine learning... ". I do not understand If you do the math, you will find that the ...
Jose_Peeterson's user avatar
1 vote
1 answer
237 views

Can the exact same split occur in subsequent trees in a gradient boosted trees model?

I am aware that there are several questions about feature/split selection in gradient-boosted trees (e.g. If a feature has already split, will it hardly be selected to split again in the subsequent ...
jokokojote's user avatar
2 votes
1 answer
116 views

Clarification on the connection between deep ensembles and bayesian neural networks

I'm not sure if I understand the relationship between deep ensembles and Bayesian interpretation well. can you tell me if i am right or wrong? Suppose I have an ensemble of L neural networks trained ...
Alucard's user avatar
  • 325
2 votes
0 answers
74 views

Any differences between Super-learner vs. Stacked generalizations vs. Stacked regressions?

I'm trying to figure out the differences between the "Super Learner" approach of Van der Laan et al. (2007), the "Stacked regressions" approach of Leo Breiman (1996) and the "...
Björn's user avatar
  • 32.2k
1 vote
1 answer
44 views

Is the practical implementation of Bootstrapping different in Statistics and Bagging Algorithms

I am learning about bagging ensemble techniques like Random Forests and the concepts of Row Sampling, Pasting, Random Subspace, and Random Patches Methods. What I understood is that bagging involves ...
tanmay's user avatar
  • 111
1 vote
0 answers
52 views

How to deal with a regression where a subset of data has more target information

I am regressing the hire price of venues. I have a dataset with lots of venues containing information on each venue. Each row is one venue and an associated price and an associated day of week from ...
Rupert Hart's user avatar
2 votes
1 answer
88 views

Understanding Stacked Generalization

I am trying to figure out how stacked generalization works? I think we train n models on the same dataset and get their class probabilities. Then these class probabilities are fed into another model. ...
deniyore's user avatar
0 votes
1 answer
63 views

Is it OK to re-sample the same patients at multiple points in time and then use the data for classification?

I'm trying to predict which patients will have a bad outcome (BO) within the next 2 weeks based on medical readings (plus some behavioral data) from the last 6 weeks. Total data timespan is 8 weeks, i....
Kyle Liaw's user avatar
1 vote
1 answer
154 views

Is data leakage a concern when using an ensemble of leave-one-out predictions?

I am new to stacking. I have a dataset with N samples and 7 tables corresponding to different data types, plus a binary label. Some tables have dozens of features, other have many thousands. I train ...
SebDL's user avatar
  • 221
4 votes
1 answer
1k views

Ensemble classifiers trained using different sets of features

Background I have a dataset, each record in this set is represented by 2 different sets of features. Let's say feature set A and feature set B. I have trained classifiers using feature set A and ...
BigTeeth's user avatar
1 vote
0 answers
274 views

Feature selection via RFE, MRMR, embeded methods and categorical features' impact

I am using ensemble-tree for regression (in Matlab) for my research. I have 22 features that includes 16 continuous (numerical) and 6 categorical variables. Categorical variables are based on time, ...
Smoa's user avatar
  • 11

1
2 3 4 5
10