Questions tagged [ensemble-learning]

In machine learning, ensemble methods combine multiple algorithms to make a prediction. Bagging, boosting and stacking are some examples.

Filter by
Sorted by
Tagged with
1 vote
0 answers
8 views

Any differences between Super-learner vs. Stacked generalizations vs. Stacked regressions?

I'm trying to figure out the differences between the "Super Learner" approach of Van der Laan et al. (2007), the "Stacked regressions" approach of Leo Breiman (1996) and the "...
  • 24k
1 vote
1 answer
18 views

Is the practical implementation of Bootstrapping different in Statistics and Bagging Algorithms

I am learning about bagging ensemble techniques like Random Forests and the concepts of Row Sampling, Pasting, Random Subspace, and Random Patches Methods. What I understood is that bagging involves ...
  • 111
0 votes
0 answers
15 views

Ensemble model with 2 disaparate feature sets

wanted to raise a theoretical discussion and ask if it's possible to make an ensemble machine learning model with 2 models trained to predict one common output variable but on entirely different ...
1 vote
0 answers
21 views

How to deal with a regression where a subset of data has more target information

I am regressing the hire price of venues. I have a dataset with lots of venues containing information on each venue. Each row is one venue and an associated price and an associated day of week from ...
2 votes
1 answer
19 views

Understanding Stacked Generalization

I am trying to figure out how stacked generalization works? I think we train n models on the same dataset and get their class probabilities. Then these class probabilities are fed into another model. ...
0 votes
0 answers
5 views

Ensemble modeling strategy

Since the ensembling model requires the individual models to be different for effectiveness, can I run two xgb models with one model metric as recall and the other one's metric as precision and ...
  • 203
0 votes
0 answers
15 views

How to get AUC and TSS values from a ensemble object in SDM?

I built a model in R combining 3 different methods using the sdm function: ...
0 votes
1 answer
28 views

Is it OK to re-sample the same patients at multiple points in time and then use the data for classification?

I'm trying to predict which patients will have a bad outcome (BO) within the next 2 weeks based on medical readings (plus some behavioral data) from the last 6 weeks. Total data timespan is 8 weeks, i....
0 votes
1 answer
17 views

Is data leakage a concern when using an ensemble of leave-one-out predictions?

I am new to stacking. I have a dataset with N samples and 7 tables corresponding to different data types, plus a binary label. Some tables have dozens of features, other have many thousands. I train ...
  • 91
3 votes
1 answer
271 views

Ensemble classifiers trained using different sets of features

Background I have a dataset, each record in this set is represented by 2 different sets of features. Let's say feature set A and feature set B. I have trained classifiers using feature set A and ...
1 vote
0 answers
32 views

Feature selection via RFE, MRMR, embeded methods and categorical features' impact

I am using ensemble-tree for regression (in Matlab) for my research. I have 22 features that includes 16 continuous (numerical) and 6 categorical variables. Categorical variables are based on time, ...
  • 11
2 votes
1 answer
66 views

Model Selection vs. Ensemble Learning

Is model selection just a specific kind of ensemble learning, where ensemble learning is loosely defined as "combining multiple models in some capacity to hopefully get an improved model"? ...
1 vote
1 answer
35 views

How to calculate $\sigma^{-1} (\frac{1}{n} \sum_{i=1}^n \sigma(x_i))$ in a numerically stable way?

Suppose I have $n$ logits $x_1, \dots, x_n$, where $n$ is not too large. They are real numbers in $(-\infty, +\infty)$ and correspond to probabilities in $(0, 1)$ via the formula $p_i = \sigma(x_i)$, ...
  • 152
0 votes
0 answers
38 views

Outputs from models (trained on different data) as inputs to another model?

Let's say I want to detect new species of fish. I have several models, each trained to recognize a different characteristic, e.g., the speed of known fish, the size of known fish, their known shapes, ...
  • 235
0 votes
0 answers
16 views

Assemble neural networks to improve performance [duplicate]

I am approaching the world of Geometric Deep Learning for the first time and I have a question, I hope someone can answer it. I am currently working on models to classify some drugs as highly active, ...
0 votes
0 answers
25 views

Usage of deep learning results as bayesian prior?

Lets consider the following situation. We are trying to estimate the parameters for a predictor where interpretability is of importance (linear regression, logistic regression etc.). When we build our ...
  • 133
0 votes
0 answers
28 views

Ensemble learning - two binary classifiers with very different weights

I have a simple ensemble composed of two binary classifiers. I've found that weighting the output probabilities as in the table below provides the best performance. Classifier Class1 Class2 Model1 ...
  • 1
2 votes
2 answers
51 views

Diversity between classifiers in ensemble learning

According to Wikipedia, Ensemble of models tends to yield better results when there is a significant diversity among the models. Many ensemble methods, therefore, seek to promote diversity among the ...
  • 47
0 votes
0 answers
49 views

Rules, theoreticial basis on selecting which machine learning model acceptable to be combined into a voting classifier

Background: Voting ensembles (hard/maximum voting, averaging/soft voting) and stack models are considered as the ensemble technique that can improve individual performance of the machine learning ...
0 votes
0 answers
130 views

How to calculate Cosine Similarity from Keras model?

I'm trying to make hybrid recommender system that recommends movies to users from Movielens dataset. Its Content part is based on Doc2Vec model from gensim library and its Collaborative Filtering part ...
1 vote
1 answer
76 views

XGBoost Objective Derivation Problem

This is the loss function of XGBoost. This is the Second-order approximation of the loss function. Note: \begin{equation} L^{(t)} \text{: cross entropy loss function.} \end{equation} \begin{equation}...
0 votes
0 answers
143 views

Bagging with SVM and Neural Networks in R with caret

I am fairly new to the bagging technique and Caret's bagControl() as well as bag() and am currently trying to build an ensemble ...
  • 1
2 votes
0 answers
88 views

Training loss goes back up but validation accuracy continues growing (XGBoost)

Using an XGBoost classifier model on a few hundred thousands rows with +/- 300 numerical features and 3,000 target classes, training with multi:softproba. Main ...
  • 529
1 vote
2 answers
42 views

How is loss propagated in stacked classifiers?

In a 2-stage stacked classifier the first model takes the input data and outputs feature vectors, which are then fed into a second model as input. The second model learns the mapping between the ...
  • 55
1 vote
1 answer
70 views

LGBM Intuition to Non-technical individuals

How can i explain LGBM to a non-technical person as it involves Trees/Ensembling and much more? Using LGBM for solving a Regression problem and how does it helps in: Better Prediction Feature ...
4 votes
2 answers
316 views

Can decision stumps have more than 2 leaves?

I understand decision stump: a shallow 1-level decision tree is often used as base-leaner in ensemble methods such as AdaBoost. What is not immediately clear to me ...
  • 231
1 vote
0 answers
42 views

Stacking neural nets with cross validation

I am trying to implement stacking model for a ML problem and having hard time figuring out the cross validation strategy. So far I have used 10-fold cross validation for all my models and would like ...
0 votes
0 answers
13 views

Bayesian Interpretation of Deep Ensembles

I was wondering if training a neural network in the deep ensemble setting can lead to a network with a posterior vs. a point estimate architecture? Recently there have been discussions over the ...
3 votes
0 answers
27 views

should trees in an ensemble be trained on samples of the same size?

I know that if bootstrap=True, then "for each tree, N samples are drawn randomly with replacement from the training set and the tree is built on this new ...
1 vote
0 answers
18 views

Does the number of estimators in an ensemble have any effect on model complexity?

First off, what I understand by "model complexity" is, roughly, the dimensionality of the parameter space. More complex means more variance (less bias) and therefore more tendency to overfit....
1 vote
1 answer
49 views

Average of t-distributed random variables

I have 10 t-distributed random variables that I'm averaging over. They are unlikely to be independent but for simplicity let's just assume that they are. Each random variable is parameterised by mean $...
  • 761
3 votes
2 answers
248 views

Why did meta-learning (or model stacking) underperform the individual base learners?

I want to use meta-learning, specifically, stacking to combine the results of two algorithms, denoted here A and B. The results of A and B correspond to the first and second columns in the dataset '...
0 votes
1 answer
50 views

How does Hard Voting in Random Forests work, if the predictors are split equally in their predictions?

I have been reading up on Hard voting classifier which is stated as a majority vote classifier. Suppose there are 4 predictors and they train on a data set with 2 outcomes (0 and 1). If 3 predictors ...
0 votes
0 answers
30 views

Model ensembling when classifiers work with different classification thresholds

I have a 2-class classification problem at hand and trained three classifiers to tackle this task. In doing so, I determined for each classifier the optimal classification threshold. For example, ...
  • 87
0 votes
1 answer
27 views

Whether can stacked generalization (stacking) further improve the performance if learner A significantly outperform B?

We know that stacking is the most popular meta-learning technique. It learns from the predictions of the base learners that learn from the training dataset. Now assuming there are two base learners, ...
0 votes
0 answers
77 views

odds ratio from Xgboost

I am using a Xgboost for a classification problem. The output is binary {0,1} and some of the input variables are categorical while the others are continuous. I would like to know if it is possible to ...
0 votes
0 answers
31 views

Comparing Two Ensemble Methods

Algorithm1 uses a single base classifier as a member of the ensemble. Suppose the size is 5 and each member in the ensemble is a Naive Bayes. The training data is shuffled/sampled (may generate a ...
1 vote
1 answer
55 views

How to choose base classifier in ensembles?

Recently, I have come through some papers, in which we could find a statement like this: "We choose Hoeffding Classifier as the base classifier, and k=15 is set for the ensemble".. According ...
0 votes
0 answers
28 views

Stacked Ensemble with Varying Weights

I have three separate models that all seek to predict the same thing per person. Each model uses a different data set with different sample sizes and then aggregates by person. For example: Model 1 is ...
0 votes
1 answer
190 views

Best way to combine the output of 2 neural networks?

I have 2 neural network models (pre-trained transformers BERT, but the input data (fine-tuning data) is different in each model) it's a binary classification task (1 or 0). Model 1 --> achieves an ...
  • 101
1 vote
0 answers
17 views

Variance analysis on boosting approachs, Is there any guarantee that boosting will not worse the weak learner variance or even get it better?

I'm looking for a theorical justification why boosting does work in pratice, I'm almost sure that this reduces the bias of their weak learners (assuming all weak learners have the same bias), but I ...
0 votes
0 answers
49 views

Why does this ensemble model score worse, when I add features via recipe function?

One of my costumers wants to get some feature engineering done in the near future. As I am using recursive ensembles from modeltime, I need to add some additional features via recipes, as ...
0 votes
0 answers
16 views

Combining multiple datasets vs multiple models in high dimensions

This question is related to this one and this one, but I was wondering about this topic in general. Imagine a setting where multiple datasets, representing different measurements, have been gathered. ...
0 votes
0 answers
81 views

What is a good method for applying grid search on ensemble models?

I built an experiment where i am studying the performance of ensemble models for a classification task. Basically, i'm comparing Random Forest with Adaboost. However, Adaboost is built with a mix of ...
  • 133
1 vote
1 answer
170 views

Normalize different binary prediction probability thresholds

I am trying to build an ensemble of three binary classifiers: A, B and C. Each one generates probabilities for the positive class. My goal is to generate a single probability for each case from the ...
  • 1,647
1 vote
2 answers
496 views

Can I compare the output probabilities of two machine learning models?

I'm sorry if this is a silly question. Suppose there are two logistic regression models $M_1$ and $M_2$ trained on the same (or similar) dataset, and their outputs of given input $x$ are $P_{M_1}(y \...
  • 133
0 votes
0 answers
38 views

What is the advantage of a mixture of experts (MoE) architecture for DNN?

Theoretically, a DNN with enough parameters can fit any training data. Thus, what is the advantage of using a mixture of experts (MoE) architecture for DNN? Is there any relevant paper about this? p.s....
0 votes
1 answer
180 views

Ensemble Model using Stacking

I learned that building an ensemble model using stacking is done by training a meta-model on the predictions of $n$ other models in order to combine the predictions and try to enhance the performance. ...
2 votes
0 answers
112 views

Model Stacking and tuning a meta-model - CV strategy?

I was hoping some of the more experienced ensemblers could help me with a couple of questions I have regarding stacking. The assumption is that we have a classic ...
0 votes
0 answers
165 views

Uncertainty Quantification in conditional VAE

I would like to collect some thoughts and references on how to quantify the uncertainties in predictions of neural network based models. In particular, I am using a conditional VAE to translate ...

1
2 3 4 5
9