# Tag Info

## New answers tagged boosting

0

If you want to use a mutlivariate approach, there are actually two approaches I'm aware of. 1.) A Vector Autoregression (VAR) learns past lags from historical sales and can still forecast. If your data is aggregated on a yearly base and you only have to forecast one year, then its possible to do it without a cumulative error, because the nearer your ...

1

I think your understanding is maturing and you now see the direct reason as to why the learning_rate parameter is also referred as shrinkage_rate. A smaller learning step size stops us from over-fitting by making our boosting process less aggressive. This "less aggression" is implemented by re-weighting our boosting weight values accordingly, as ...

1

The reason for using 'stumps' in boosting but full-height trees in random forests is to do with how the aggregation and fitting is done. In random forests, the trees in the ensemble are fitted independently to independent bootstrap samples, so any error caused by growing the trees too far is independent for each tree and tends to cancel out in the ensemble ...

1

Assuming with short decision trees, you mean trees with only a single split, the AdaBoost model will only capture main effects of predictors. If a variable is never selected for splitting, it thus does not have a main effect on the response variable (over and above the effects of the other variables that were selected for splitting). Variable selection ...

0

Uh, oh, old question... ! Linear model A linear regression model usually has one or more of three purposes: Effect estimation Testing hypotheses Prediction While there is no multicollinearity assumption behind the classic normal linear model, strong multicollinearity is problematic regarding the first two items: Effect estimates are unnatural due to ...

2

It is just using a linear model with l1 and l2 regularization as its base learner rather than a decision tree. Here is a similar Q&A: Difference in regression coefficients of sklearn's LinearRegression and XGBRegressor . So it will be different than other linear models because it is optimized slightly differently but more-so you are boosting it which ...

1

My (limited) understanding is that you can make finding the best-variance split into an $O\left( N \right)$ operation. Note: You are asking about variance, which applies to continuous variables, and not something like gini score (aka not gini coefficient) which is used for categorical ones. Setup: Lets say you have a the following case: library(pracma) ...

0

If you think about the gradient-boosted CART (aka an atom of gbm), the model is "boxes"(1). This (data): Is represented by a (CART fit) as this: Each leaf-tip is a mean. Each split is perpendicular to an axis. It is trying to adjust the box bounds and the mean height to minimize error in representation. From the above, you see the motivation ...

1

As you have a common starting state for all cases and multiple final states (deaths from different causes), this is a classic competing-risks situation as explained in the R multi-state survival vignette. The Aalen-Johnson model explained in Chapter 2 of the vignette is a non-parametric description of the probability of being in each state as a function of ...

3

Using the binary log-loss classification as an objective is a good move in this situation (and in most situations). We might want to point Optuna (or our general hyper-parameter search framework) to minimise the Brier score of the predictions if we care about how much the probabilities might be off; the AUC-ROC is a ranking score, it is better than F1-score ...

2

Responding to your comments, and bringing it back to my original comment: Line 4 of Friedman's algorithm can be viewed as "finding the best function $g \in \mathcal{G}$ that fits the pseudo-residuals via minimizing $\sum_i [\tilde{y}_i - g(x_i)]^2$," with $\mathcal{G}$ containing all functions of the form $g(\cdot) := \beta h(\cdot; a)$. Once you ...

1

ROC-AUC always tracks with precision, recall, and f1-score, and since you are obtaining AUC values of 0.8 with everything else near 0.48-0.5, you are obviously overfitting. You didn't say anything about number of objects, features, folds, or anything about how you are testing objects left out of training. It seems that you are also employing one data set, ...

1

Because a bootstrap by definition involves drawing a random sample (with replacement) which has the same sample size as the original dataset, i.e., $n$ objects. On average, when randomly drawing $n$ objects with replacement from a sample of $n$ objects, the probability that an object will not be selected is $(1-1/n)^n=\exp(-1)=0.368$. Thus, the probability ...

0

Given that both the f1-score and PR AUC are very low even for the prevalence of ~0.45%, it can not be deduced if the limitations are imposed by the nature of the data or the model (features plus the algorithm used). In order to build a better understanding and to resolve the issue, I would suggest to break the problem into two parts: Build a model that ...

1

Whether or not the importances of gradient boosting are strongly influenced by randomness depends on the hyper-parameter configuration of the gradient boosting model. A gradient boosted model which uses random subsampling of features (or other randomized components) will estimate feature importances which vary to a greater or lesser degree upon repeated ...

Top 50 recent answers are included