Understanding the behavior of a neural network when extrapolated

Question

I trained a Multilayer Perceptron to predict a variable Y based on a set of predictors. Then I decided to test it on unseen data outside of the training range. I am aware of (some of) the implications of extrapolating machine learning models, and how ANN specifically can lead to crazy extrapolations. Nevertheless, my experiment requires this step, and I believe the shape the MLP produces when out of range is not necessarily the issue.

The issue, as seen in the partial dependence plot below, is that I would expect the extrapolation (red curve) to follow the down slope of the training curve (black curve). Instead, what we see is an almost identical curve, but translated to the right of the training curve. I would appreciate any insights on why this is happening, or any comments suggesting there is one or more flaws on my logic. Lastly, it would be interesting to hear thoughts on how to achieve this "extension" of the training curve on to the extrapolation curve.

            def create_model():
                model = Sequential()
                model.add(Dense(200, input_dim=len(X_train.columns))) 
                model.add(Activation('relu'))
                model.add(Dropout(0.1))
    
                model.add(Dense(200))
                model.add(Activation('relu'))
                model.add(Dropout(0.1))
    
                model.add(Dense(200))
                model.add(Activation('relu'))
                model.add(Dropout(0.1))
                
                model.add(Dense(200))
                model.add(Activation('relu'))
                model.add(Dropout(0.1))
                
                model.add(Dense(1, activation='linear'))
                # compile the keras model
                model.compile(loss='mean_absolute_error', optimizer=tf.keras.optimizers.Adam(0.001), metrics=['mean_squared_error','mean_absolute_error'])
                return model
                    
            model_rf = Pipeline([
                ('scaler', StandardScaler()),
                ('estimator', KerasRegressor(model=create_model, epochs=200, batch_size= 1024, verbose=1))

Answer 1

If a neural net is built with ReLu units, then its asymptotic behaviour is necessarily linear. No training can change this.

More generally, no machine learning with a finite training set can train asymptotic behaviour. So extrapolations always reflect a priori assumptions, not training.

Answer 2

It seems you are using your fit outside of the expected range. You don't want to extrapolate. Ever. Period. End of story.

Anything extrapolated can not be trusted. The extrapolation results will depend on the coefficients you get when you fit and the functional form of the fit (your model). If your extrapolated data do not behave this way, you will get nonsense (or something unexplainable-- as you have seen)

Please see:

https://www.sciencedirect.com/science/article/pii/S2772415821000110

Section I shows how one can be tricked into thinking their training data cover their desired outcomes.

Section II gives an easy algorithm for people to use to quickly determine if their data point is extrapolated and how much of the possible outcome space is an extrapolation (based on training data).

Section III shows a great example of a great neural network fit, but what happens when you use it as a prediction and the inputs are out of range.

Section IV shows how much data you actually need to fit a deep neural network. I would focus on sections III and IV. Do not trust if out of range!

For those interested, the full citation:

Siegel, Adam. "A parallel algorithm for understanding design spaces and performing convex hull computations." Journal of Computational Mathematics and Data Science 2 (2022): 100021.

Answer 3

After some weeks of testing, I think I finally figured out the solution for the issue above. And it is rather basic.

First, it's good to recall the two datasets used: the historical one, which was used for training and validation, and the future one, for tests purpose. Being the datasets timeseries, they were detrended. However, they were corrected by their mean values, as it is the standard approach in the field. Again, it is a basic thing that I did not realise before, but this leads to a significant bias between training data and tests data.

The solution was to adjust the detrending of the future dataset to have a mean value equal to the historical dataset. This way, with the mean values equal, the extrapolation became smooth and more similar to what I would have expected in the first place. Figure below illustrates the behaviour.