I'm using these ensemble algorithms for learning, and they are performing quite well. Now, I need to explain them to my peers and myself. Copied from the page, it says
Transform your features into a higher dimensional, sparse space. Then train a linear model on these features.
How are these models transforming the features into higher dimensional, sparse space?
In the paragraph below, I get the
fit an ensemble of trees on the training set. The next sentence about how
each leaf of each tree is assigned a fixed arbitrary feature index in a new feature space is absolutely not clear; where and how did the new feature space come about?
Furthermore, why are the leaf indices one-hot encoded?
First fit an ensemble of trees (totally random trees, a random forest, or gradient boosted trees) on the training set. Then each leaf of each tree in the ensemble is assigned a fixed arbitrary feature index in a new feature space. These leaf indices are then encoded in a one-hot fashion.
In the last paragraph, it just seems as if the original features are turned off and on.
Each sample goes through the decisions of each tree of the ensemble and ends up in one leaf per tree. The sample is encoded by setting feature values for these leaves to 1 and the other feature values to 0.
From what I can understand, it just seems that there are a bunch of trees where each leaf node turns off/on features from the original space; this understanding seems to me to be reducing the feature space not projecting it to higher dimensions.
In the example with gradient boosted tree with logistic regression, it seems the algorithm is learning trees to turn on/off features (feature selection), then applying logistic regression to classify on a subset of the features. Of course, this understanding is completely against what's being stated.
Are there any more in-depth documentations on these techniques/algorithms?