Questions tagged [ensemble-learning]

In machine learning, ensemble methods combine multiple algorithms to make a prediction. Bagging, boosting and stacking are some examples.

Filter by
Sorted by
Tagged with
0 votes
0 answers
19 views

Rules, theoreticial basis on selecting which machine learning model acceptable to be combined into a voting classifier

Background: Voting ensembles (hard/maximum voting, averaging/soft voting) and stack models are considered as the ensemble technique that can improve individual performance of the machine learning ...
user avatar
0 votes
0 answers
27 views

How to calculate Cosine Similarity from Keras model?

I'm trying to make hybrid recommender system that recommends movies to users from Movielens dataset. Its Content part is based on Doc2Vec model from gensim library and its Collaborative Filtering part ...
user avatar
1 vote
1 answer
35 views

XGBoost Objective Derivation Problem

This is the loss function of XGBoost. This is the Second-order approximation of the loss function. Note: \begin{equation} L^{(t)} \text{: cross entropy loss function.} \end{equation} \begin{equation}...
user avatar
0 votes
0 answers
37 views

Bagging with SVM and Neural Networks in R with caret

I am fairly new to the bagging technique and Caret's bagControl() as well as bag() and am currently trying to build an ensemble ...
user avatar
  • 1
1 vote
0 answers
20 views

Training loss goes back up but validation accuracy continues growing (XGBoost)

Using an XGBoost classifier model on a few hundred thousands rows with +/- 300 numerical features and 3,000 target classes, training with multi:softproba. Main ...
user avatar
  • 509
1 vote
2 answers
35 views

How is loss propagated in stacked classifiers?

In a 2-stage stacked classifier the first model takes the input data and outputs feature vectors, which are then fed into a second model as input. The second model learns the mapping between the ...
user avatar
  • 55
1 vote
1 answer
36 views

LGBM Intuition to Non-technical individuals

How can i explain LGBM to a non-technical person as it involves Trees/Ensembling and much more? Using LGBM for solving a Regression problem and how does it helps in: Better Prediction Feature ...
user avatar
4 votes
2 answers
226 views

Can decision stumps have more than 2 leaves?

I understand decision stump: a shallow 1-level decision tree is often used as base-leaner in ensemble methods such as AdaBoost. What is not immediately clear to me ...
user avatar
  • 231
1 vote
0 answers
34 views

Stacking neural nets with cross validation

I am trying to implement stacking model for a ML problem and having hard time figuring out the cross validation strategy. So far I have used 10-fold cross validation for all my models and would like ...
user avatar
0 votes
0 answers
7 views

Bayesian Interpretation of Deep Ensembles

I was wondering if training a neural network in the deep ensemble setting can lead to a network with a posterior vs. a point estimate architecture? Recently there have been discussions over the ...
user avatar
3 votes
0 answers
25 views

should trees in an ensemble be trained on samples of the same size?

I know that if bootstrap=True, then "for each tree, N samples are drawn randomly with replacement from the training set and the tree is built on this new ...
user avatar
1 vote
0 answers
12 views

Does the number of estimators in an ensemble have any effect on model complexity?

First off, what I understand by "model complexity" is, roughly, the dimensionality of the parameter space. More complex means more variance (less bias) and therefore more tendency to overfit....
user avatar
1 vote
1 answer
43 views

Average of t-distributed random variables

I have 10 t-distributed random variables that I'm averaging over. They are unlikely to be independent but for simplicity let's just assume that they are. Each random variable is parameterised by mean $...
user avatar
  • 749
2 votes
2 answers
131 views

Why did meta-learning (or model stacking) underperform the individual base learners?

I want to use meta-learning, specifically, stacking to combine the results of two algorithms, denoted here A and B. The results of A and B correspond to the first and second columns in the dataset '...
user avatar
0 votes
1 answer
33 views

How does Hard Voting in Random Forests work, if the predictors are split equally in their predictions?

I have been reading up on Hard voting classifier which is stated as a majority vote classifier. Suppose there are 4 predictors and they train on a data set with 2 outcomes (0 and 1). If 3 predictors ...
user avatar
0 votes
0 answers
15 views

Model ensembling when classifiers work with different classification thresholds

I have a 2-class classification problem at hand and trained three classifiers to tackle this task. In doing so, I determined for each classifier the optimal classification threshold. For example, ...
user avatar
  • 87
0 votes
1 answer
22 views

Whether can stacked generalization (stacking) further improve the performance if learner A significantly outperform B?

We know that stacking is the most popular meta-learning technique. It learns from the predictions of the base learners that learn from the training dataset. Now assuming there are two base learners, ...
user avatar
0 votes
0 answers
45 views

odds ratio from Xgboost

I am using a Xgboost for a classification problem. The output is binary {0,1} and some of the input variables are categorical while the others are continuous. I would like to know if it is possible to ...
user avatar
0 votes
0 answers
24 views

Comparing Two Ensemble Methods

Algorithm1 uses a single base classifier as a member of the ensemble. Suppose the size is 5 and each member in the ensemble is a Naive Bayes. The training data is shuffled/sampled (may generate a ...
user avatar
1 vote
1 answer
44 views

How to choose base classifier in ensembles?

Recently, I have come through some papers, in which we could find a statement like this: "We choose Hoeffding Classifier as the base classifier, and k=15 is set for the ensemble".. According ...
user avatar
0 votes
0 answers
17 views

Stacked Ensemble with Varying Weights

I have three separate models that all seek to predict the same thing per person. Each model uses a different data set with different sample sizes and then aggregates by person. For example: Model 1 is ...
user avatar
0 votes
1 answer
38 views

Best way to combine the output of 2 neural networks?

I have 2 neural network models (pre-trained transformers BERT, but the input data (fine-tuning data) is different in each model) it's a binary classification task (1 or 0). Model 1 --> achieves an ...
user avatar
  • 101
0 votes
0 answers
11 views

Variance analysis on boosting approachs, Is there any guarantee that boosting will not worse the weak learner variance or even get it better?

I'm looking for a theorical justification why boosting does work in pratice, I'm almost sure that this reduces the bias of their weak learners (assuming all weak learners have the same bias), but I ...
user avatar
0 votes
0 answers
43 views

Why does this ensemble model score worse, when I add features via recipe function?

One of my costumers wants to get some feature engineering done in the near future. As I am using recursive ensembles from modeltime, I need to add some additional features via recipes, as ...
user avatar
0 votes
0 answers
10 views

Combining multiple datasets vs multiple models in high dimensions

This question is related to this one and this one, but I was wondering about this topic in general. Imagine a setting where multiple datasets, representing different measurements, have been gathered. ...
user avatar
0 votes
0 answers
50 views

What is a good method for applying grid search on ensemble models?

I built an experiment where i am studying the performance of ensemble models for a classification task. Basically, i'm comparing Random Forest with Adaboost. However, Adaboost is built with a mix of ...
user avatar
  • 133
1 vote
1 answer
73 views

Normalize different binary prediction probability thresholds

I am trying to build an ensemble of three binary classifiers: A, B and C. Each one generates probabilities for the positive class. My goal is to generate a single probability for each case from the ...
user avatar
  • 1,557
1 vote
2 answers
210 views

Can I compare the output probabilities of two machine learning models?

I'm sorry if this is a silly question. Suppose there are two logistic regression models $M_1$ and $M_2$ trained on the same (or similar) dataset, and their outputs of given input $x$ are $P_{M_1}(y \...
user avatar
  • 113
0 votes
0 answers
33 views

What is the advantage of a mixture of experts (MoE) architecture for DNN?

Theoretically, a DNN with enough parameters can fit any training data. Thus, what is the advantage of using a mixture of experts (MoE) architecture for DNN? Is there any relevant paper about this? p.s....
user avatar
0 votes
1 answer
76 views

Ensemble Model using Stacking

I learned that building an ensemble model using stacking is done by training a meta-model on the predictions of $n$ other models in order to combine the predictions and try to enhance the performance. ...
user avatar
2 votes
0 answers
57 views

Model Stacking and tuning a meta-model - CV strategy?

I was hoping some of the more experienced ensemblers could help me with a couple of questions I have regarding stacking. The assumption is that we have a classic ...
user avatar
0 votes
0 answers
79 views

Uncertainty Quantification in conditional VAE

I would like to collect some thoughts and references on how to quantify the uncertainties in predictions of neural network based models. In particular, I am using a conditional VAE to translate ...
user avatar
  • 123
0 votes
0 answers
21 views

Method for selecting the best peforming classification model

I have trained 4 classification models specifically: logistic regression, a decision tree classifier, a random forest classifier and xgboost. I have used the validation set to compute the ROC AUC ...
user avatar
  • 1
6 votes
0 answers
255 views

How shap values behave in terms of multicollinearity in Trees, Ensemble, GradientBoosting and GAM/Boosting

I set up an experiment with these 8 Regressor Methods: sklearn package DecisionTreeRegressor, RandomForestRegressor, ExtraTreesRegressor, GradientBoostingRegressor other packages CatBoostRegressor ,...
user avatar
1 vote
0 answers
28 views

Why a meta ensemble of ensemble models works well?

Why is ensembling the predictions of an ensemble model effective in improving test error? Is there any literature that discusses the theory behind this? In fact, ensembling GBMs or LGBMs is widely ...
user avatar
1 vote
2 answers
48 views

How to combine predictions from ensemble learning

Suppose I have three models, model-1, model-2, model-3 for binary classification. Suppose model-1 has $a_1$ accuracy, model-2 has $a_2$ accuracy, model-3 has $a_3$ accuracy. For some test data, model-...
user avatar
  • 13
2 votes
1 answer
124 views

can we trust feature importance of a poor model

Usually when training ensemble learning algorithms or a regression model we calculate feature importance and make conclusion that feature with highest feature importance has largest affect on y-...
user avatar
  • 201
0 votes
0 answers
78 views

Confusion with AdaBoost.M1 algorithm

I'm having my head buried in the AdaBoost.M1. Are there yet some variations of AdaBoost.M1? I ask this because I read variations ...
user avatar
  • 231
3 votes
2 answers
112 views

Where is Boosting applied in Gradient boosting techniques?

In boosting, the primary idea is to re-adjust weights of training instances, so that subsequent models learn how to fit difficult-to-classify samples. From Wikipedia Boosting (Machine Learning): ...
user avatar
2 votes
1 answer
39 views

Hunting "Sweetspots" In Predictive Models

I have three separate predictive models that each accept a set of inputs about an upcoming event and produce a probability of that event being true. I have domain knowledge that leads me to suspect ...
user avatar
2 votes
1 answer
111 views

Can Bayesian Model Averaging be Optimal when the Hypothesis Space does not contain the true hypothesis?

I am utterly confused. I have been reading about the optimality of Bayes classifier and Bayesian model averaging all the time, but when I try to dig deeper, I just get more confused. On the one hand, ...
user avatar
14 votes
2 answers
1k views

What if there is no true data-generating process?

I've been engaging in a number of forecasting efforts recently, and have rediscovered a well-known truth: That combinations of different forecasts are generally better than the forecasts themselves. ...
user avatar
  • 2,587
3 votes
2 answers
147 views

Why in the stacking of scikit-learn the estimators are fitted on the whole training data?

In chapter 7 of "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow", the first step of stacking method is spliting the train data into two subsets. The first subset is used ...
user avatar
  • 33
0 votes
1 answer
43 views

Combining DecisionTreeClassifiers

I have an array of sklearn.tree._classes.DecisionTreeClassifier classifiers that are used in a boosting algorithm, so the final classifier is a weighted sum of these individual trees. The problem is ...
user avatar
1 vote
0 answers
32 views

Difference between ensembling models and using one model as a feature in the second model?

I have 2 models - one is a generative model (a model which assets the parameter of some distribution for each individual object needs to be predicted, and does not learn from any data), and the second ...
user avatar
  • 47
1 vote
1 answer
31 views

Approximating the posterior and learning the distribution over the weights after training

I am familiar with the methods in variational inference in which after training we have access to the distribution over the network's weights. This is necessary for estimating epistemic uncertainty. ...
user avatar
0 votes
1 answer
46 views

How to evaluate ensemble models using independent data

I am creating species distribution models using the popular R package biomod2. I am using biomod2 functionality to create ensemble models. Ensemble models combine individual models based on their ...
user avatar
  • 101
2 votes
1 answer
150 views

Fitting Stacking Classifer when Underlying Models Use Different Feature Subsets

I’m looking to create a stacking classifier based on 3 underlying algorithms. I have already performed feature selection on each of the 3, and each returned a slightly different subset of features for ...
user avatar
  • 21
2 votes
1 answer
48 views

What does the phrase "ensemble methods" mean in machine learning? What exactly is going on here?

I know this might be a newbie question, but I'm trying to make sense of this sentence. I understand that these are several algorithms that together give a good result. But I don't understand how this ...
user avatar
  • 121
0 votes
0 answers
23 views

Combined probability from multiple predictors

I built three different models that estimate the outcome of basketball games. These matches are played between Team A and Team B, and each model gives in output the value ...
user avatar

1
2 3 4 5
9