Is there any possibility to obtain regression equation (y=...) by using tree algorithms (e.g.random forest, gradient boost or xgboost) ? As I understand, the target that we attempt to use these algorthims is only to learn which predictors are important or not. If that is the case, the only way to achieve a regression model is to use a OLS or PLSR methods by utilizing the most influential predictors coming from tree algortihms. Am I correct? What is your advice on using tree algorithms for those who wants to get a response model like me?
5
-
1$\begingroup$ In it's current form this question is very unclear. Please have a look at it again and try to express it in a clearer manner. E.g. , what do you mean by "how does this model predict my response data values regardless of being based on a mathematical expression" ? Or in the first paragraph, what do you mean by "regardless of sticking to any mathematical expression" ? $\endgroup$– deemelCommented Jul 30, 2018 at 8:55
-
1$\begingroup$ Could you specify which ML model are you talking about? There are many different ones, and they depend on different math principles. $\endgroup$– Jan KukackaCommented Jul 30, 2018 at 9:04
-
$\begingroup$ I made my question simple as it is. Let's talk specifically. For example, tree algorithm (random forest regression) gives us a correlation coefficient between predicted and true value. But, as I know it does not give us a regression equation. Without constructing a model like in statistical learning, how does it give us an estimation on true value? And if tree algorithms does not give us a regression equation, why do they include "'regression term" in their names such as random forest regression? $\endgroup$– ŞAHİN SARRAFİCommented Jul 30, 2018 at 9:18
-
$\begingroup$ This question is very broad, and I believe you would profit from reading an introductory level textbook. We have a helpful list of free statistical textbooks. If afterwards you still have more specific questions, then please do ask them here. If you already have read such a textbook, please edit your question to make it more specific. Thank you! $\endgroup$– Sycorax ♦Commented Jul 31, 2018 at 0:39
-
$\begingroup$ I revised my question. I know the regression model and how it works. I took many courses on it during my master. So, I am quite experienced on statistical modeling and machine learning in practical as an applied engineer. But the theory behing the machine learning intermingle with statistical learning. Some concepts (like regression) are using by both ML and Stat.learning. By the way, the main question that brings myself to this point is the current (revised) question about tree algorithms and random forest regression. Thanks for your advices. $\endgroup$– ŞAHİN SARRAFİCommented Jul 31, 2018 at 10:27
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
You can turn a single tree structure into a regression-like equation with the use of indicator functions to indicate the different nodes of the tree. That is, you can make an equation with the dependent variable on the left side and the independent variables on the right. These equations get cumbersome quickly, if the tree is at all complex.
As far as I can see, the same ought to be possible with methods such as random forests, but I am not expert on them and can't say for sure.
answered Jul 31, 2018 at 11:15
$\begingroup$
$\endgroup$
The XGBoost portion of the question is answered here: How does gradient boosting calculate probability estimates?
Peter Flom's outline, that you can use boolean logic to represent the decision tree output, is correct. The same procedure generalizes to random forests.
In the case that each tree votes (that is, makes a binary decision), another perspective is to think of each tree as producing a 1-hot sparse vector. An ensemble of $T$ such trees produces a $T$ hot sparse vector. The average of that vector is the average over each ensemble member.
In the case that each tree produces a proportional score in $(0,1)$, the result is a sparse vector with one float, and you can acquire average scores in the same manner.
answered Jul 31, 2018 at 19:50