Questions tagged [ensemble-learning]

In machine learning, ensemble methods combine multiple algorithms to make a prediction. Bagging, boosting and stacking are some examples.

Filter by
Sorted by
Tagged with
0
votes
1answer
22 views

Ensemble Model using Stacking

I learned that building an ensemble model using stacking is done by training a meta-model on the predictions of $n$ other models in order to combine the predictions and try to enhance the performance. ...
2
votes
0answers
16 views

Model Stacking and tuning a meta-model - CV strategy?

I was hoping some of the more experienced ensemblers could help me with a couple of questions I have regarding stacking. The assumption is that we have a classic ...
0
votes
0answers
20 views

Uncertainty Quantification in conditional VAE

I would like to collect some thoughts and references on how to quantify the uncertainties in predictions of neural network based models. In particular, I am using a conditional VAE to translate ...
0
votes
0answers
18 views

Method for selecting the best peforming classification model

I have trained 4 classification models specifically: logistic regression, a decision tree classifier, a random forest classifier and xgboost. I have used the validation set to compute the ROC AUC ...
4
votes
0answers
50 views

How shap values behave in terms of multicollinearity in Trees, Ensemble, GradientBoosting and GAM/Boosting

I set up an experiment with these 8 Regressor Methods: sklearn package DecisionTreeRegressor, RandomForestRegressor, ExtraTreesRegressor, GradientBoostingRegressor other packages CatBoostRegressor ,...
1
vote
0answers
15 views

Why a meta ensemble of ensemble models works well?

Why is ensembling the predictions of an ensemble model effective in improving test error? Is there any literature that discusses the theory behind this? In fact, ensembling GBMs or LGBMs is widely ...
1
vote
2answers
40 views

How to combine predictions from ensemble learning

Suppose I have three models, model-1, model-2, model-3 for binary classification. Suppose model-1 has $a_1$ accuracy, model-2 has $a_2$ accuracy, model-3 has $a_3$ accuracy. For some test data, model-...
2
votes
1answer
40 views

can we trust feature importance of a poor model

Usually when training ensemble learning algorithms or a regression model we calculate feature importance and make conclusion that feature with highest feature importance has largest affect on y-...
0
votes
1answer
34 views

Confusion with AdaBoost.M1 algorithm

I'm having my head buried in the AdaBoost.M1. Are there yet some variations of AdaBoost.M1? I ask this because I read variations ...
2
votes
2answers
78 views

Where is Boosting applied in Gradient boosting techniques?

In boosting, the primary idea is to re-adjust weights of training instances, so that subsequent models learn how to fit difficult-to-classify samples. From Wikipedia Boosting (Machine Learning): ...
2
votes
1answer
37 views

Hunting "Sweetspots" In Predictive Models

I have three separate predictive models that each accept a set of inputs about an upcoming event and produce a probability of that event being true. I have domain knowledge that leads me to suspect ...
3
votes
1answer
92 views

Can Bayesian Model Averaging be Optimal when the Hypothesis Space does not contain the true hypothesis?

I am utterly confused. I have been reading about the optimality of Bayes classifier and Bayesian model averaging all the time, but when I try to dig deeper, I just get more confused. On the one hand, ...
13
votes
2answers
986 views

What if there is no true data-generating process?

I've been engaging in a number of forecasting efforts recently, and have rediscovered a well-known truth: That combinations of different forecasts are generally better than the forecasts themselves. ...
3
votes
2answers
63 views

Why in the stacking of scikit-learn the estimators are fitted on the whole training data?

In chapter 7 of "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow", the first step of stacking method is spliting the train data into two subsets. The first subset is used ...
0
votes
1answer
30 views

Combining DecisionTreeClassifiers

I have an array of sklearn.tree._classes.DecisionTreeClassifier classifiers that are used in a boosting algorithm, so the final classifier is a weighted sum of these individual trees. The problem is ...
1
vote
0answers
30 views

Difference between ensembling models and using one model as a feature in the second model?

I have 2 models - one is a generative model (a model which assets the parameter of some distribution for each individual object needs to be predicted, and does not learn from any data), and the second ...
1
vote
1answer
29 views

Approximating the posterior and learning the distribution over the weights after training

I am familiar with the methods in variational inference in which after training we have access to the distribution over the network's weights. This is necessary for estimating epistemic uncertainty. ...
0
votes
1answer
25 views

How to evaluate ensemble models using independent data

I am creating species distribution models using the popular R package biomod2. I am using biomod2 functionality to create ensemble models. Ensemble models combine individual models based on their ...
1
vote
1answer
51 views

Fitting Stacking Classifer when Underlying Models Use Different Feature Subsets

I’m looking to create a stacking classifier based on 3 underlying algorithms. I have already performed feature selection on each of the 3, and each returned a slightly different subset of features for ...
2
votes
1answer
42 views

What does the phrase "ensemble methods" mean in machine learning? What exactly is going on here?

I know this might be a newbie question, but I'm trying to make sense of this sentence. I understand that these are several algorithms that together give a good result. But I don't understand how this ...
0
votes
0answers
22 views

Combined probability from multiple predictors

I built three different models that estimate the outcome of basketball games. These matches are played between Team A and Team B, and each model gives in output the value ...
0
votes
0answers
49 views

How does resampling in AdaBoost (exactly) work?

Overall, I like to think that I understand how AdaBoost works, i.e., fitting a weak learner, calculating the error, calculating the confidence / amount of say of the learner, updating the sample ...
7
votes
2answers
116 views

Intutition of why Bootstrap aggregating reduces overfitting?

Can somebody give me a non-mathematical intuition why Bootstrap aggregating reduces overfitting? From my point of view, we are not providing any additional information, we are not really enlengthen ...
0
votes
0answers
31 views

What is the actual Mechanism of Boosting for building the Models?

I couldn't able to find the proper step by step procedure to understand the Boosting Mechanism, how does it build the models and the data used to build it. So, I have gone through tutorials which led ...
0
votes
0answers
17 views

Ensemble model predicts negative and extreme values for a non-negative response variable

I am using SuperLearner() in R to create an ensemble model composed of GLM, GAM, randomForest, gradient boosting model, and multivariate regression spline models. My response variable is tree density, ...
0
votes
0answers
12 views

Ensemble of boolean predictors

Lets say we have a 0/1 predictor, that is right in p % of cases. If we ensemble N of those uncorrelated predictors, what is (an elegant solution for) their accuracy? (The ensemble's answer is the most ...
0
votes
0answers
11 views

How can I perform ensemble subset selection based on the OOB error?

As we all know, selecting the optimal random forest based on the out-of-bag (OOB) error is an efficient way to determine the best model without an additional validation set. However, it appears that ...
1
vote
1answer
170 views

What methods can be used to overcome the tie-breaking when using majority voting in ensemble?

What methods can be used to overcome the tie-breaking when using majority voting in ensemble? I read that Weighted majority voting can help; however, it wasn't effective on the dataset I am using in ...
1
vote
1answer
177 views

What's the purpose of learning rate in sklearn AdaBoost implementation

We know that sklearn's implemenation of AdaBoost algorithm uses DecisionTreeClassifier as the base learner. Conceptually, ...
0
votes
0answers
22 views

Classification Problem using multiclass features input and ensemble methods

I am working on a classification problem. I am applying tree-ensemble methods (Histogram-Based Gradient Boosting and Random Forest) and evaluating premutation importance in order to understand ...
0
votes
0answers
19 views

Gradient Boosting vs XGBoost key differences

I know XGBoost minimize a regularized loss function instead GB (gradient boosting) but I dont know how trees grow, it would be a simple fit to estimate G/H? where G is first derivate with respect to ...
1
vote
0answers
31 views

Which library implements AdaBoost.NC? [closed]

The sklearn implementation of AdaBoost uses the AdaBoost.SAMME variant of the algorithm. Is there a known python-based library the uses the ...
2
votes
0answers
37 views

Is Gradient Boosting a particular algorithm of Any Boost?

Any Boost is describe as follows: I haven't ever seen a inner product as stop criteria in Gradient Boosting algorithm. In Any Boost the weak learner is a function $f_{t+1}$ that maximizes inner ...
2
votes
1answer
53 views

Random forest that aggregates by taking the maximum over the trees instead of taking the average

I want to make a Random forest that aggregates by taking the maximum over the decision trees instead of taking the average. By default Sklearn is taking the average, and I couldn't find how to change ...
2
votes
0answers
17 views

Applying ensambled train models on test data vs ensambling test predictions

Consider we are fitting linear model to solve classification/regression problem on the given train data. What would be a better strategy to follow from the below mentioned ones to evaluate the model ...
0
votes
0answers
12 views

Where does the Bias and Var reduction in ensemble learning come from?

A boosting model as follows seeks to have low bias than a single model: $f(x)=\sum_{i=1}^{N}\beta_mh_m(x)$ Where $h_m$ is any weak learner to be trained sequentially. A bagging model tries reduce ...
0
votes
0answers
24 views

To avoid over-fitting an ensemble model, should one penalize at both the ensemble and the individual model levels?

If you have an ensemble method that is similar to your lower-level modeling methods, such as a weighted average of a set of penalized linear models (LASSO, ridge, or whatever), with the weights chosen ...
0
votes
0answers
12 views

Is a regression tree closed under scaling?

I've tried proof that a regression tree is closed under scaling but I'm not sure if I've meant right. Please go through. Let $D^n$ be a feature space, a regression tree can be viewed as function that $...
15
votes
2answers
2k views

Can I (justifiably) train a second model only on the observations that a previous model predicted poorly?

Say I commit the following sins while building a predictive model: I take my dataset and split it into four subsets: Three for training (Train_A, Train_B, and Train_C) and one for validation. I ...
1
vote
1answer
89 views

Does Wikipedia explain gradient boosting in wrong way?

Wikipedia's geral Gradient Boosting is: Friedman's Gradient Boosting is: Why wikipedia's gradient boosting fit h_m through pseudo-residuals while friedman uses line 4 to fit h? My question is not ...
2
votes
2answers
273 views

Does Adaboost ensemble use bootstrapping?

I am reading about boosting methods in the book Elements of statistical learning. In page 339 they describe the Ada boost algorithm as I understand the general idea behind it: Give more weight to ...
2
votes
0answers
13 views

Gradient Boosting algorithm

What is beta for? Why not just fit h_m (get a_m) without this beta? since h_m is an estimator for pseudo-residuals
0
votes
1answer
70 views

How to interpret the direct comparison of Continuous Rank Probability Score (CRPS) and Mean Absolute Error (MAE)?

Say I have a trained Random Forest (RF) consisted of $m$ decision trees and I am interested to estimate $y$ from $t_1$ to $t_n$. The good thing about RF is that I have an ensemble of estimators and a ...
0
votes
0answers
21 views

How to build an ensemble for different set of features?

I have a dataset in which part of the features have more data than the other part, and for avoiding to build a full data set with a small amount of data, I'd like to build two models, each one ...
1
vote
1answer
44 views

Is multicollinearity ever an issue in ensemble learning?

Suppose I have two models, A and B, and suppose B takes the output of A as one of its features. Now suppose that both models use at least some of the same features. Is there a potential ...
0
votes
0answers
20 views

Methods to reduce regression underestimate and overestimate

I'm new to a project and need to reduce the underestimate & overestimate cases in a regression problem. So far haven't gained enough domain knowledge. Underestimate is less tolerable than ...
0
votes
1answer
41 views

Is it necessary to retrain a random forest instead of removing trees when comparing accuracy between different numbers of trees?

I have a train data set and a validation set using which I wish to optimize the hyperparameter that is the number of trees in a binary classification random forest (scikit-learn). (As Sycorax ...
0
votes
0answers
12 views

What is the best programmatic way to find the best ensemble model in python, as to which models are best suited to which portions of the data

Generally in ensemble modelling as dataset is being segregated into multiple portions where each portion is being trained on a particular model, what is the programmatic way to determine which model ...
2
votes
1answer
56 views

What is the difference beetwen Random Forest and Random Subspace Method?

Is the only difference that the Random Forest enforces the use of decision trees as a base learner and use bootstrap sampling?
1
vote
0answers
25 views

Different types of Ensemble learning methods

I've been reading and searching information about different types of Ensemble learning methods however I am a bit confused and want to make sure my understanding is correct. Below is graph of how I ...

1
2 3 4 5
9