Regression forests are basically random forests, however used for regression. They basically use the same framework as decision forests use for classification with a few parts exchanged.

Two of these parts are:

  1. How to calculate the information gain when deciding how to split at a node?
  2. How do we make a prediction when we reach a leaf?

An interesting report (however 150 pages strong) concerning this topic is the following: http://research.microsoft.com/apps/pubs/default.aspx?id=155552

This report touches the above mentioned problems, I however do not really understand the possible solutions.

1) Page 52, Appendix A. They define the continuous formulation of information gain to be: $ I_j = \sum_{v \in S_j} log(|\Lambda_y(v)|) - \sum_{i \in \{L,R \}} (\sum_{v \in S_j^i} log(|\Lambda_y(v)|)) $. As I dont really know where else to ask: Does anyone know what || is in this case? How would you implement a function to compute I?

2) When we reach a leaf we have a number of data points reaching this leaf, say of the form (x,y). From my understanding we would now fit a function f(x) to these data points (e.g. by least squares) and would then use f(x) to predict an y for a future testing point x. In the report (page 50) they talk about a probabilistic linear fitting, that returns a conditional distribution p(y|x). How could that be done? I dont understand how we can compute a probability density function here.

I hope anyone is familiar with the above mentioned report and might be able to help :) Thank you!

  1. Information Gain is just a metric used to choose the attribute on which the data has to be split at the node. One important point to note is that, while Information Gain is the metric used in classification, Standard Deviation is used as the metric in case of Regression Forest. While the attribute with the highest Information Gain is chosen as the attribute to split the data on in case of classification, the attribute with the highest decrease in Standard Deviation is used in case of regression.
  2. After the regression tree is built, each leaf node in the tree might contain multiple samples from the training set. The average of the target variables of this leaf node is the output for any test sample that follows this decision path.

You can check out my personal blog for more information about regression trees: Decision Trees, Random Forests and XGBoost

Hope that answers your question.

  • 2
    $\begingroup$ Please don't spam your blog. $\endgroup$ Feb 27 '17 at 13:53
  • 4
    $\begingroup$ Please note if you want to promote your own product/blog you must disclose your affiliation, otherwise your answer may be flagged as spam. Please read How to not be a spammer $\endgroup$ Feb 27 '17 at 13:54
  • 2
    $\begingroup$ Welcome to our site, Sujay. You will note this community is quick to react to evidence of possible abuse. I would like to emphasize that we value experts and we understand that being able to refer to their own work is part of what makes someone an expert. The only objections being raised in the comments you're getting about alleged "spam" concern the lack of attribution. You can address those simply by stating that the blog is yours anywhere you reference it here. $\endgroup$
    – whuber
    Feb 27 '17 at 14:43
  • $\begingroup$ @DavidPostill Thank you for those pointers. I had no intentions of promoting my blog. I answered the two questions posted in the question. Since I had explored regression trees in detail in my blog, I thought it would be a helpful guide to the asker. Since it would be too lengthy to include the content of the blog here, I added a link. It was just a reference to the asker. $\endgroup$ Feb 28 '17 at 15:47
  • $\begingroup$ @whuber Thank you for the tip. Would stating that the blog is mine in the answer (like the edit) suffice or is there some other procedure I need to follow? $\endgroup$ Feb 28 '17 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.