4
$\begingroup$

I haven't used decision trees in the past, and I'm looking into them now.

With regression trees, I am wondering if we are technically performing classification instead of regression. We train our decision tree using training data that have continuous outputs, but the regression itself has a finite number of nodes and thus a finite number of outputs. So when you feed in an example into the tree, there's a finite number of predictions it can make. Doesn't that effectively make the regression problem a classification problem ?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
6
$\begingroup$

Consider linear regression with a single parameter. It will always predict one number, namely the sample mean of the outcome on which it was trained. Is this regression even though it can only give one output?

If you subscribe to the definition of regression as a learning problem in which the output is quantitative rather than categorical (or perhaps more strongly, a learning problem in which the predictions could possibly belong to some infinte set rather than a finite set), then trees are performing regression. They are capable of learning to output continuous values rather than prediction category membership (well...they can do that too, but assume we're only talking about regression trees).

Frankly, what we call what the tree does is not really important. If you want to insist it is classification, be my guest. Though I suspect you'll see some eyebrows raised when you do.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I don't really get the linear regression analogy. For a single input, it clearly has a unique output, but linear regression itself can generate an infinite number of outputs $\in \mathbb{R}$ whereas a regression tree has an output $\in T$, where $|T|$ is the number of leaves in the tree. $\endgroup$
    – 24n8
    Sep 20, 2020 at 17:22
  • 6
    $\begingroup$ The distinction you are making is between a linear function and a piecewise constant function -- but that's neither fundamental nor conceptually relevant to regression. See the hits at stats.stackexchange.com/search?q=regression+definition for more about this. $\endgroup$
    – whuber ♦
    Sep 20, 2020 at 17:40
  • 2
    $\begingroup$ @anonuser01 consider simple linear regression with single binary feature $y = \beta_0 + \beta_1 X + \varepsilon$. in such case, it can predict either $\beta_0$ for $X=0$, or $\beta_0 + \beta_1$ for $X = 1$, so also two outcomes. Regression is about predicting numerical outcomes. $\endgroup$
    – Tim ♦
    Sep 20, 2020 at 17:48
  • 1
    $\begingroup$ @whuber The "piecewise constant function" part clicked for me. $\endgroup$
    – 24n8
    Sep 20, 2020 at 18:25
  • $\begingroup$ @Tim In your example I agree that I would categorize that as a regression problem even though there are only 2 outcomes. But when you say regression is about prediction "numerical" outcomes, are you implying that classification isn't? Classification could also predict numerical outcomes. $\endgroup$
    – 24n8
    Sep 20, 2020 at 18:26
0
$\begingroup$

The term "classification" is generally used to refer to situations where the output is from a small number of options that do not have any structure, such as an inherent order (the term is sometimes still used in cases where the output does have a structure, but usually it doesn't).

The definition of "regression" is often given as meaning that the output is continuous, but there are a finite number of values that a float variable can take, yet we consider it to be "continuous". There is a point at which a large enough number of possible outputs, with a clear numerical structure of order, etc., is considered to be "regression".

For instance, if SAT score is considered to be a predictive model of future college grades, it would generally be considered to be more of a regression than a classification model, even though there are a finite number of possible scores. Netflix's percentage match is regression, even though there are only 101 different possible percentages. Something along the lines of the Homeland Security Advisory System is debatable: it has a small number of categories, but those categories are ordered. Making it even more complicated, classification models often are derived from a regression model. For instance, if you're training a model to detect "cat" versus "noncat", you likely are going to calculate a "catlike" score, and output "cat" if it exceeds a threshold.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.