Search Results

So, trees are great but why not a random forest or some other ensembling technique rather than boosting? … Neural nets also can overfit so it is not a good candidate for boosting but there are are still relevant algos such as the resnet architecture which is theoretically related to boosting, see here. …

But, for example with gradient boosting we utilize HEAVY regularization such as a learning rate and subsampling procedures. …

So it will be different than other linear models because it is optimized slightly differently but more-so you are boosting it which provides further regularization in linear models unlike when you boost …

Yes, weak learners are absolutely required for boosting to be really successful. That is because each boosting round for trees actually results in more splits and a more complicated model. … Although, linear models won't be able to give you the best accuracy when boosting (typically). You mention kaggle and deep trees. …

The boosting algorithm which takes each round's residuals and trains the next model on these 'psuedo' residuals can be applied to basically any other model. … To your other point, the boosting and mlp methods follow the same routine for classification or regression we just do transformations to allow for classification. …

Yeah I wouldn't say that trees are not affected by it but that they are 'generally' more robust than most other methods like a linear model. This is due to how they handle features by assessing each a …

What others have said about weak learners being preferred is 100% true, but the core reason why trees or GAMs would be useful to boost is that by boosting them you gain a more complicated model. … A simple linear regression is weak but boosting it won't do too much because the output is still a simple linear regression, maybe with a slightly different coefficient. …

I would dummy it up but it kind of depends, for example years of education and predicting income. You can and should dummy it in a normal linear regression but for a tree it makes sense to leave it b …

Since boosting is iteratively learning from the past model it can overfit so the learning rate is a simple way to control for that. … As a side note I have seen it heavily used much more aggressively in gradient boosting. …

Anytime you would use data that you wouldn't have at forecast time while testing then you introduce data leakage. This occurs when you use the lags and you want to forecast more than 1 period for t-1 …

ThymeBoost just combines gradient boosting and time-series decomposition so essentially the decision tree is itself a boosted tree which is interesting. …