XGBoost: does manipulating the sample make it "extrapolate"?

Question

Suppose I want to perform time series forecasting with XGBoost. I understand that tree-based models cannot extrapolate. However, the time series I am working with is stationary (no trend or obvious seasonality + the ADF test gives a p-value of basically zero on the train sample). My problem is that since I divide my data into train and test subsamples, some observations (outliers) in the test set cannot be predicted by the model, because it has not seen such low/high values on the train set. The model gives flat lines near the bounds.

• I am aware of stacking/hybrid models (for example XGBoost + LinReg) to allow for extrapolation. However, as far as I know, this only fixes the problem of extrapolating trends or seasonal patterns, whereas my concern is about data range or outliers. If I were to fit an XGBoost model, get predictions $\hat{y}_t$ and then model the residuals $y_t-\hat{y}_t$ with some other stacked model, then I am not aware of any model that would be good at predicting just some sudden peaks with all other values being close to zero (so, basically, white noise with sudden peaks)
• I also know that there exists an option in XGBoost to choose gblinear as the booster instead. However, in my particular case the range for my data is by definition in $(0,+\infty)$ (I am predicting volatility), so this would allow the model to get to negative values. Also, I have tried fitting this model anyway and the fit was terrible, much worse than the default gbtree
• The only thing I came up with was to change, say, the first two observations in the train set to some made up outliers (for example, I set $y_1=0$ and $y_2=100$), which the model will definitely never see again to try and force this range on it, so it can at least consider values close to these. Even though visually I did not see the difference, all the metrics I used (like RMSE and MAE) improved quite a lot just from this simple fix. However, the model still flatlines at the same spots as it used to

My questions are: are there any other techniques to try and fix this problem? Is my solution even legit at all? Does it allow for "extrapolation"?
Any suggestions are greatly appreciated

Answer 1

If there is no information that allows your system to learn when spikes will occur, it will not predict spikes, no matter what you do. This is just another way of saying that you always have unpredictable noise, and your system does not (and should not) try to predict it, because unpredictability is, roughly speaking, the definition of noise.

Therefore, your predictions will always vary less than your actuals. See a few links here: Canonical duplicate for "Why do predictions vary less than observations?"

On predictability and irreducible noise, this may be helpful: How to know that your machine learning problem is hopeless?