4
$\begingroup$

We know, roughly speaking, the algorithm of decision tree is like an approximation with piecewise constants.

For example, suppose we have a function $f$ from $[0,1]$ to $\mathbb R$. Then we can use decision tree to approximate it, i.e. using piecewise constant functions to approximate it. The trained model can tell us an output for any given input $a\in [0,1]$. But it cannot predict an output for an input $b\in [2,3]$.

Now, let us take the $x$-axis as time and the $y$-axis as stock price. Suppose the time interval is $[1,100]$, where every integer means one day. Then by doing decision tree, we will have a model. But this model can only be used to predict the stock price of any time in $[1,100]$, not for the future. That's, we cannot use it to predict the stock price at time $101$, which is outside $[1,100]$.

But there are people using decision tree to predict the stock prices. Why can they make sense of the reasoning? [I tried to read their sample codes but without clear comments along with them I couldn't really understand the logics.]

Thanks for helping me make sense of the logics.

$\endgroup$
1
  • 1
    $\begingroup$ you don't use time, but eg the previous stock price value: If stock price yesterday is < 50, predict 25 else if stock price yesterday <100 predict 75 $\endgroup$
    – seanv507
    Commented Oct 17, 2022 at 15:07

2 Answers 2

8
$\begingroup$

As also pointed out in the comments section, time is usually not in your feature list when you do time series prediction with tree-based algorithms.

Typical features are target values at previous time instants, other hand crafted features like weekend, evening indicators etc.

$\endgroup$
2
  • $\begingroup$ Thanks for the quick response. Does it mean, we take {target values at previous time instants, other hand crafted features} as inputs and take {target values for now} as outputs, and then we train the decision tree with these input-output pairs? If we do such pairing, we need to throw away the target value at the first time instant because it doesn't have any previous time instant, right? $\endgroup$
    – Sam Wong
    Commented Oct 17, 2022 at 15:17
  • 2
    $\begingroup$ That’s right and yes you should start from the time instant such that all your features can be calculated. $\endgroup$
    – gunes
    Commented Oct 17, 2022 at 15:28
8
$\begingroup$

More generally (ignoring the specifics of the problem), decision trees (and related algorithms like gradient boosted decision trees, random forest etc.) will make predictions for numeric feature values outside of the range of feature values seen at training. The splits for branches of the tree are usually done as "variable #1 < X" vs. "variable #1 >= X" (whether it's < and >=, or <= and > doesn't really matter) and such splits are only done at values that lie within the range of values seen during training, so any value higher than values seen during training will essentially be treated like the highest value seen during training. Similarly, the a value lower than the lowest value seen during training will be treated like the lowest value seen during training.

In other words, no matter how far outside the range of values seen during training, the predictions will no longer change beyond the highest (or lowest) feature value seen during training.

This has also been summarized as "decision trees don't extrapolate", which is to say that even if there is e.g. a blatantly obvious linear relationship of feature to outcome, a decision tree will not continue to extrapolate it, but rather "stay flat with its predictions".

In contrast, other models (e.g. linear regression, neural networks etc.) will "happily" extrapolate patterns beyond the training data. Of course, this may often be a very, very wrong thing to do.

In general, extrapolation beyond the training data involves a lot of assumptions and is usually best done if one truly understands the underlying data generating process instead of modeling it empirically. An example where it is to some extent possible is e.g. predicting drug concentrations for individuals that are much larger or smaller than those in early clinical trials of a drug from pharmacokinetic model (that tends to model the process of absorption, distribution and excretion of a drug taking into account some characteristics of patients). If one does not have such a situation, then perhaps we sometimes need to predict something, but should then be much more cautious about relying on the prediction.

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.