Questions tagged [ensemble-learning]

In machine learning, ensemble methods combine multiple algorithms to make a prediction. Bagging, boosting and stacking are some examples.

Filter by
Sorted by
Tagged with
2
votes
1answer
79 views

Can Bayesian Model Averaging be Optimal when the Hypothesis Space does not contain the true hypothesis?

I am utterly confused. I have been reading about the optimality of Bayes classifier and Bayesian model averaging all the time, but when I try to dig deeper, I just get more confused. On the one hand, ...
13
votes
2answers
956 views

What if there is no true data-generating process?

I've been engaging in a number of forecasting efforts recently, and have rediscovered a well-known truth: That combinations of different forecasts are generally better than the forecasts themselves. ...
3
votes
2answers
40 views

Why in the stacking of scikit-learn the estimators are fitted on the whole training data?

In chapter 7 of "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow", the first step of stacking method is spliting the train data into two subsets. The first subset is used ...
0
votes
1answer
27 views

Combining DecisionTreeClassifiers

I have an array of sklearn.tree._classes.DecisionTreeClassifier classifiers that are used in a boosting algorithm, so the final classifier is a weighted sum of these individual trees. The problem is ...
1
vote
0answers
29 views

Difference between ensembling models and using one model as a feature in the second model?

I have 2 models - one is a generative model (a model which assets the parameter of some distribution for each individual object needs to be predicted, and does not learn from any data), and the second ...
1
vote
1answer
28 views

Approximating the posterior and learning the distribution over the weights after training

I am familiar with the methods in variational inference in which after training we have access to the distribution over the network's weights. This is necessary for estimating epistemic uncertainty. ...
0
votes
1answer
22 views

How to evaluate ensemble models using independent data

I am creating species distribution models using the popular R package biomod2. I am using biomod2 functionality to create ensemble models. Ensemble models combine individual models based on their ...
1
vote
1answer
29 views

Fitting Stacking Classifer when Underlying Models Use Different Feature Subsets

I’m looking to create a stacking classifier based on 3 underlying algorithms. I have already performed feature selection on each of the 3, and each returned a slightly different subset of features for ...
2
votes
1answer
42 views

What does the phrase "ensemble methods" mean in machine learning? What exactly is going on here?

I know this might be a newbie question, but I'm trying to make sense of this sentence. I understand that these are several algorithms that together give a good result. But I don't understand how this ...
0
votes
0answers
21 views

Combined probability from multiple predictors

I built three different models that estimate the outcome of basketball games. These matches are played between Team A and Team B, and each model gives in output the value ...
0
votes
0answers
18 views

How does resampling in AdaBoost (exactly) work?

Overall, I like to think that I understand how AdaBoost works, i.e., fitting a weak learner, calculating the error, calculating the confidence / amount of say of the learner, updating the sample ...
4
votes
1answer
59 views

Intutition of why ensemble learning reduces overfitting?

Can somebody give me a non mathematical intuition why ensemble learning reduce overfitting? From my point of view, we are not providing any additional information, we are not really enlengthen the ...
0
votes
0answers
30 views

What is the actual Mechanism of Boosting for building the Models?

I couldn't able to find the proper step by step procedure to understand the Boosting Mechanism, how does it build the models and the data used to build it. So, I have gone through tutorials which led ...
0
votes
0answers
17 views

Ensemble model predicts negative and extreme values for a non-negative response variable

I am using SuperLearner() in R to create an ensemble model composed of GLM, GAM, randomForest, gradient boosting model, and multivariate regression spline models. My response variable is tree density, ...
0
votes
0answers
12 views

Ensemble of boolean predictors

Lets say we have a 0/1 predictor, that is right in p % of cases. If we ensemble N of those uncorrelated predictors, what is (an elegant solution for) their accuracy? (The ensemble's answer is the most ...
0
votes
0answers
11 views

How can I perform ensemble subset selection based on the OOB error?

As we all know, selecting the optimal random forest based on the out-of-bag (OOB) error is an efficient way to determine the best model without an additional validation set. However, it appears that ...
1
vote
1answer
104 views

What methods can be used to overcome the tie-breaking when using majority voting in ensemble?

What methods can be used to overcome the tie-breaking when using majority voting in ensemble? I read that Weighted majority voting can help; however, it wasn't effective on the dataset I am using in ...
1
vote
1answer
71 views

What's the purpose of learning rate in sklearn AdaBoost implementation

We know that sklearn's implemenation of AdaBoost algorithm uses DecisionTreeClassifier as the base learner. Conceptually, ...
0
votes
0answers
20 views

Classification Problem using multiclass features input and ensemble methods

I am working on a classification problem. I am applying tree-ensemble methods (Histogram-Based Gradient Boosting and Random Forest) and evaluating premutation importance in order to understand ...
0
votes
0answers
17 views

Gradient Boosting vs XGBoost key differences

I know XGBoost minimize a regularized loss function instead GB (gradient boosting) but I dont know how trees grow, it would be a simple fit to estimate G/H? where G is first derivate with respect to ...
1
vote
0answers
25 views

Which library implements AdaBoost.NC? [closed]

The sklearn implementation of AdaBoost uses the AdaBoost.SAMME variant of the algorithm. Is there a known python-based library the uses the ...
2
votes
0answers
33 views

Is Gradient Boosting a particular algorithm of Any Boost?

Any Boost is describe as follows: I haven't ever seen a inner product as stop criteria in Gradient Boosting algorithm. In Any Boost the weak learner is a function $f_{t+1}$ that maximizes inner ...
2
votes
1answer
53 views

Random forest that aggregates by taking the maximum over the trees instead of taking the average

I want to make a Random forest that aggregates by taking the maximum over the decision trees instead of taking the average. By default Sklearn is taking the average, and I couldn't find how to change ...
2
votes
0answers
16 views

Applying ensambled train models on test data vs ensambling test predictions

Consider we are fitting linear model to solve classification/regression problem on the given train data. What would be a better strategy to follow from the below mentioned ones to evaluate the model ...
0
votes
0answers
12 views

Where does the Bias and Var reduction in ensemble learning come from?

A boosting model as follows seeks to have low bias than a single model: $f(x)=\sum_{i=1}^{N}\beta_mh_m(x)$ Where $h_m$ is any weak learner to be trained sequentially. A bagging model tries reduce ...
0
votes
0answers
20 views

To avoid over-fitting an ensemble model, should one penalize at both the ensemble and the individual model levels?

If you have an ensemble method that is similar to your lower-level modeling methods, such as a weighted average of a set of penalized linear models (LASSO, ridge, or whatever), with the weights chosen ...
0
votes
0answers
12 views

Is a regression tree closed under scaling?

I've tried proof that a regression tree is closed under scaling but I'm not sure if I've meant right. Please go through. Let $D^n$ be a feature space, a regression tree can be viewed as function that $...
15
votes
2answers
2k views

Can I (justifiably) train a second model only on the observations that a previous model predicted poorly?

Say I commit the following sins while building a predictive model: I take my dataset and split it into four subsets: Three for training (Train_A, Train_B, and Train_C) and one for validation. I ...
1
vote
1answer
85 views

Does Wikipedia explain gradient boosting in wrong way?

Wikipedia's geral Gradient Boosting is: Friedman's Gradient Boosting is: Why wikipedia's gradient boosting fit h_m through pseudo-residuals while friedman uses line 4 to fit h? My question is not ...
2
votes
2answers
181 views

Does Adaboost ensemble use bootstrapping?

I am reading about boosting methods in the book Elements of statistical learning. In page 339 they describe the Ada boost algorithm as I understand the general idea behind it: Give more weight to ...
2
votes
0answers
10 views

Gradient Boosting algorithm

What is beta for? Why not just fit h_m (get a_m) without this beta? since h_m is an estimator for pseudo-residuals
0
votes
1answer
52 views

How to interpret the direct comparison of Continuous Rank Probability Score (CRPS) and Mean Absolute Error (MAE)?

Say I have a trained Random Forest (RF) consisted of $m$ decision trees and I am interested to estimate $y$ from $t_1$ to $t_n$. The good thing about RF is that I have an ensemble of estimators and a ...
0
votes
0answers
14 views

How to build an ensemble for different set of features?

I have a dataset in which part of the features have more data than the other part, and for avoiding to build a full data set with a small amount of data, I'd like to build two models, each one ...
1
vote
1answer
33 views

Is multicollinearity ever an issue in ensemble learning?

Suppose I have two models, A and B, and suppose B takes the output of A as one of its features. Now suppose that both models use at least some of the same features. Is there a potential ...
0
votes
0answers
17 views

Methods to reduce regression underestimate and overestimate

I'm new to a project and need to reduce the underestimate & overestimate cases in a regression problem. So far haven't gained enough domain knowledge. Underestimate is less tolerable than ...
0
votes
1answer
35 views

Is it necessary to retrain a random forest instead of removing trees when comparing accuracy between different numbers of trees?

I have a train data set and a validation set using which I wish to optimize the hyperparameter that is the number of trees in a binary classification random forest (scikit-learn). (As Sycorax ...
0
votes
0answers
11 views

What is the best programmatic way to find the best ensemble model in python, as to which models are best suited to which portions of the data

Generally in ensemble modelling as dataset is being segregated into multiple portions where each portion is being trained on a particular model, what is the programmatic way to determine which model ...
2
votes
1answer
45 views

What is the difference beetwen Random Forest and Random Subspace Method?

Is the only difference that the Random Forest enforces the use of decision trees as a base learner and use bootstrap sampling?
0
votes
0answers
18 views

Different types of Ensemble learning methods

I've been reading and searching information about different types of Ensemble learning methods however I am a bit confused and want to make sure my understanding is correct. Below is graph of how I ...
0
votes
1answer
32 views

Does Excess Kurtosis Signal Non-Ergodicity?

I have been reading a lot about ergodicity and the main principle behind it seems pretty simple actually. Based on my understanding, something is ergodic if the time average and the ensemble average ...
0
votes
0answers
16 views

What does it tell me when my ensemble learners (classifier trees) are just as good with only one split?

Basically new to decision trees and ensemble classifiers. Looking for guidance based on what I am seeing. My goal really isn't to use a decision tree. It is to do a simple binary (A/B) ...
1
vote
0answers
51 views

What is a reasonable number of splits (maximum) for a ensemble classifier?

I am trying to use an ensemble classifier (honing in on Matlab fitcensemble). I've also explored using a single decision tree as well as tree bagging (Matlab fitctree, TreeBagger) Simple binary (A/B) ...
0
votes
0answers
172 views

Confidence interval of ensemble prediction

I have created $N$ neural network models of the same architecture (with hyperparameters $XYZ$), each with random initializations for model weights and trained within the same workflow. These models ...
1
vote
0answers
18 views

How to combine several machine learning models trained with the same target but different sets of predictors

The book Hands-On Machine Learning with R gives an overview of model stacking, but goes on to say that the same training set shall be used for the models. The vignette of caretEnsemble also stresses ...
2
votes
1answer
40 views

Is it a good idea to use a linear model (like logistic regression) to generate new features for a non linear model (like random forest)? [duplicate]

The setting is a 2-class classification problem. We have too many features, some of them not very informative and with many zeros. We are thinking in ways of selecting the best features, and PCA (in ...
0
votes
0answers
61 views

How to tune an weighted voting ensemble method?

I am working on kidney cancer patients' data with 5 unbalanced labels. These codes are contained of Normalization, Oversampling on Feature Engineering part. A list of 9 ordinary Machine Learning ...
3
votes
1answer
64 views

in which part is used the splitting criteria in AdaBoost?

I have been reading the original article about AdaBoost and by comparing with other reading material it has come some doubts about this model. Please feel free to correct me if in any part I am wrong. ...
3
votes
1answer
31 views

Ensemble vs statistical power

In regression modelling, I've seen two schools of thought: ensemble model vs. focus on statistical power (using one model). Proponents of ensemble models (i.e., bagging) argue that: Suppose $\...
1
vote
0answers
21 views

How to sample a joint posterior given multiple models?

Consider having two models, $m_1$ and $m_2$, for a set of data $x$, each model has associated parameters $\theta_1$, for model one, and $\theta_2$ for model two, (not necessarily the same dimension). ...
1
vote
0answers
36 views

Gradient boosting doubts

I am new to the concept of Gradient Boosting and i have a few doubts related to it. It will be helpful if some one can explain them. 1) Gradient boosting is gradient descent in functional space As ...

1
2 3 4 5
9