Questions tagged [ensemble]

In machine learning, ensemble methods combine multiple algorithms to make a prediction. Bagging, boosting and stacking are some examples.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Who first coined the term Ensemble Learning?

Who first introduced/invented the "Ensemble Techniques"? Is there any published research paper that first used that term?
1
vote
0answers
58 views

What's a word meaning “drawn from the same distribution”?

I'm comparing results from ensemble data assimilation experiments where prior ensemble members are either drawn from the same (multivariate Gaussian) distribution as a "true" variable, or ...
0
votes
0answers
23 views

Stacking and Ensembling methods in Data Science

Recently it seems that stacking and ensembling methods have become more popular, and using these methods can give better results than using a single algorithm. My question is: What are the reasons, ...
0
votes
0answers
13 views

Sanity check: Ensemble-based sequential state inference with different parameters (Ensemble Kalman Filter)

I have recently been invited to review a publication which employs the Ensemble Kalman Filter (EnKF) for the sequential inference of dynamic state variables $x_{1,...,t}=(x_1,...,x_t)$. In the study, ...
1
vote
1answer
30 views

How should I decide the decision-threshold in a classification built using stacked ensemble classifier

I am attempting a stacked ensemble model to achieve binary classification for the first time. Should the decision threshold be One for which I receive max F1 score (assuming F1 score drives the ...
1
vote
1answer
29 views

Modeling Expected Value Using Quantile Regression as an Ensemble

I'm trying to find a primer on a topic that I'm sure must have been studied, but I can't find anything on. Suppose we'd like to do regression in a supervised learning setting to learn the expected ...
0
votes
1answer
20 views

Bootstrapping with strong classifiers - vs simulated annealing insights

Consider the following bootstrapping scheme, common with weak classifiers: 0. Given training data N samples by P predictors sample the training data with replacement k times (for a large k) fit ...
4
votes
2answers
80 views

Is there a “piecewise linear fitting” for logistic regression?

For regression problem we can fit the data with a piecewise linear function (Linear Splines). Is there a "piecewise linear fitting" for binary classification? Is that using spline basis ...
0
votes
0answers
47 views

How to calculate gradient for custom objective function in xgboost for FFORMA

I'm trying to build an implementation of the Feature-based Forecast Model Averaging approach in Python (https://robjhyndman.com/papers/fforma.pdf). However, I'm sort of stuck on computing the gradient ...
1
vote
0answers
12 views

Combining models trained on different shapes of input to predict same output

Sorry in advance if this is obvious I have two features, one of shape (6075) and the other is (366). I've trained two models: ...
1
vote
1answer
55 views

Bagging classifier vs RandomForestClassifier [duplicate]

Is there a difference between using a bagging classifier with base_estimaton=DecisionTreeClassifier and using just the RandomForestClassifier? This question refers to models from python library called ...
1
vote
1answer
20 views

How to measure variance that is due to random noise in training data

I would like to measure the variance of a binary classification model (deep neural network). Say the performance metric of choice is f1-score. There are two sources of variance that I can think about: ...
0
votes
0answers
25 views

Gradient Boosting and neural networks

Is there any Python package that implements a boosted neural network ? Any pointer is appreciated. A sample reference about the boosting and NN can be this one.
0
votes
0answers
8 views

Ensemble error can never be lower than best base learner

A common ensemble approach is ensemble averaging, where predictions from individual base learners are averaged (individual predictions are weighted equally to form the combined prediction). If the ...
0
votes
0answers
13 views

Level of Agree-ness in ensemble models

I am trying to analyze the output of multiple models (as in ensemble method), classifying a multiclass dataset. I would like to study the behavior of the models compared to each other when they are ...
0
votes
0answers
5 views

Voting ensemble rationale and ensemble weights

what is the rationale for averaging multiple base models' predictions? One base learner might work towards reducing variance, another might work towards reducing bias, etc, but is there a source ...
0
votes
0answers
21 views

Lasso for Ensemble Learning, base learner selection

In ensemble learning, we average the predictions of multiple base learners (e.g. SVM + ANN + Linear regression). Instead of taking the mean of the individual base models' predictions, can lasso be ...
0
votes
0answers
17 views

Majority voting error rate with binomial distribution

We all know that the ensemble error rate for majority voting rule follows binomial distribution (see picture): where p is the error probability of single classifier. However, I have hard time ...
0
votes
0answers
33 views

How to treat large chunks of missing data when ROC-AUC scoring is used

I have a dataset with a majority of features for about one-third of both train and test data are missing. E.g. I have values A, B, and C for 66% of my data and only C's for the rest of the data. The ...
0
votes
0answers
14 views

Stacking model: is finding best threshold necessary?

After performing a stacking model (with rstudio), is it necessary to choose the best threshold for it? In general after finding the best model among all the fitted models , you have to choose the ...
1
vote
0answers
72 views

Neural network doesn't converge but has good performance

I have a sequence (> 100 million) of symbols and several models predict the next symbol. To combine these predictions I'm using stacked generalization with a multilayer perceptron trained with online ...
0
votes
1answer
27 views

Why stacking use strong learner as base learner?

I am wondering why stacking uses strong learners as base learners. How to understand it from expectation and variance way, or bias and variance way?
0
votes
0answers
56 views

Stacking using layer-1 models predictions on test set

I am new to Data Science and have been studying the methods of stacking to find out if it can meet the following fact, but I did not find or understand evidence that it can or cannot work. Let's ...
0
votes
0answers
15 views

What does it mean when combining two features sets gives a worse result than just one of them?

I’ve created a 2 binary classifiers for lung cancer patients dead/alive 2 years after diagnosis. One is CNN trained on pathology images And the other is a RF trained on clinical data (age, sex, stage,...
0
votes
0answers
21 views

Generating ML models on the fly

I have the following scenario, given a k>=100, a set of features, a set of type of models, eg. SVR, ExtraTreeRegressor, etc: I want to generate a set of ...
1
vote
0answers
47 views

Order of features for gridsearch and model fitting

Assuming that the same columns (i.e., features) are used for hyperparameter tuning and model fitting, and ensemble models are used for modeling (e.g., Random forest or XGboost), then does the order of ...
1
vote
1answer
121 views

Is this kind of stacked ensemble method prone to over-fitting?

I am working on a stacked ensemble method. I trained three classifiers as my first-layer models and one Logistic Regression as my second layer model. I then stacked both the first-layer models and ...
4
votes
1answer
47 views

What does it mean when we say the estimators need to be independent when using Ensemble Methods in Machine Learning?

In collective learning (ensemble methods) we need the estimators to be Independent/ uncorrelated from one another. Do I understand correctly, that this means we need to draw the data samples without ...
0
votes
0answers
13 views

Finding the distribution of a quantity using ensemble

In physics, we have the concept of ensemble average, which we use a lot in statistical physics. For example, ergodic property states that the time average of a statistical quantity is equal to its ...
2
votes
1answer
57 views

Calculate minimum accuracy for a boosting algorithm

Suppose, you are working on a binary classification problem. And there are 3 models each with 70% accuracy. If you want to ensemble these models using majority voting. What will be the minimum ...
2
votes
0answers
55 views

Disadvantages of “moving window ensemble” approach?

Assuming online/incremental training is not available for a particular algorithm, and assuming that you have a stream of training data that may or may not change over time (eg log data), what are the ...
0
votes
0answers
12 views

Ensemble-based inference: How to deal with non-convergent ensemble members?

Assume that I want to use an ensemble-based sequential Bayesian inference (or swarm-based optimization) algorithm which relies on information provided by each ensemble member. For example, each sample ...
0
votes
0answers
7 views

Interpolating Regression Models Across Geographical Space

I have 50 timeseries datasets from 50 cities across the United States (1 for each city). The timeseries are of different lengths (they are daily timeseries anywhere from 3-30 years long), but they are ...
1
vote
0answers
22 views

class-specific neural networks vs overall network

Let's say I am working in a classification setting, like MNIST for instance, and imagine that I have 10 neural networks which are all slightly different and each perform very well on one unique class. ...
0
votes
0answers
47 views

Depth of trees in GBM: what does interaction.depth = 1 mean?

As far as I know, GBM does ensembling on weak learners which can be trees with one split. I have already read this post, but I could not find the answer for my question. My question is that what ...
3
votes
1answer
660 views

Is exponential loss function the only reason for AdaBoost being adaptive algorithm?

Main concept of AdaBoost is that on each iteration algorithm learns what samples were difficult to classify and increases weights of these samples, while decreasing weights of those that were easy to ...
3
votes
1answer
79 views

Train an ensemble of neural networks on different datasets, what is the best way to scale the inputs?

I'm training an ensemble of three neural networks using l-BFGS method for regression. Each neural network is trained on a sub-dataset that is randomly sampled from a large dataset. Since the sub-...
0
votes
0answers
37 views

Ensemble learning: Efficiently stacking Neural Network model

I want to use ensemble learning for a classification task. I built three models. One of them is a neural network, which takes an hour to train. I want to use model stacking. I do not have the ...
0
votes
0answers
45 views

Isolation Forest Numerical Example

I'm looking for a proper numerical example to understand Isolation Forests Algorithm correctly. I've read the paper : https://github.com/mgckind/iso_forest/blob/master/icdm08b.pdf, but I want to ...
0
votes
0answers
10 views

What's the meaning of building classifiers for each class in binary classification?

The question arises when I'm using DistributedRandomForest from the H2O package and find the ...
1
vote
0answers
155 views

How to create a test set in stacking when doing cross validation

I am using Weka to implement stacking with k-fold cross validation. As I understand, we first divide our dataset in to k folds, then we use k-1 folds for training and 1 fold for testing. This ...
1
vote
0answers
42 views

What is the most appropriate way to aggregate two identical linear models fit with different data?

Suppose I have a linear model y ~ Xb, and I split my observations into multiple X's X1, X2, X3 etc. What is the most appropriate way to aggregate the separate models y1 ~ X1b1, y2 ~ X2b2 to produce ...
2
votes
0answers
16 views

How to optimize hyperparameters in stacked model?

I was wondering whether somebody could explain how to optimize hyperparameters for the base learners and meta algorithm when stacking? In many tutorials they seem to be plucked out of thin air! ...
0
votes
2answers
438 views

Why is my stacked model worse than my base models? [closed]

I'm learning stacking and start with the approach outlined in Introduction stacking I've plotted the data: I first would like to check if my algorithm is correct (see below): So I basically ...
1
vote
0answers
29 views

Forecasting time series data using EEMD based SVM?

Splitting of Dataset: Dataset = Train1 + Test1 EEMD(Train1) = train1 + test1 I am forecasting on time series data("Dataset") using SVM. First I found the Intrinic Mode Function(IMF) of time series ...
1
vote
1answer
294 views

AdaBoost learning rate calculation

I saw the following in a Random Forrest calculation. My understanding of logarithms is not intuitive, I always have to look them up. It was asked: ...
0
votes
0answers
31 views

Will out of bag error always be 100% for classes of size 1 using random forests

My understanding is that given a sample data point $x_i$ the OOB prediction for this point will be calculated using only trees in the ensemble that do not contain $x_i$ in their bootstrap sample. Thus ...
1
vote
1answer
191 views

Is Stratified K Fold CV Needed when Estimator implements Balanced Class Weight?

I am working on a classification task with an imbalanced dataset. I am using Sklearn's ensemble RandomForestClassifier and set its class weight to Balanced. My question is, when I then GridSearch it, ...
1
vote
1answer
342 views

How to visualize proximity score in Random Forests

For a Random Forest, we can construct a N x N (where N is the number of data points) proximity matrix ...
1
vote
0answers
105 views

Assigning probabilities to ensemble experts (classification)

Suppose we have a set of experts which predict on a data set, and the true labels are also given. I would like to find out the probabilities for combining predictions of separate experts. So the ...

1
2 3 4 5
7