Understanding the behavior of a neural network when extrapolated I trained a Multilayer Perceptron to predict a variable Y based on a set of predictors. Then I decided to test it on unseen data outside of the training range. I am aware of (some of) the implications of extrapolating machine learning models, and how ANN specifically can lead to crazy extrapolations. Nevertheless, my experiment requires this step, and I believe the shape the MLP produces when out of range is not necessarily the issue.
The issue, as seen in the partial dependence plot below, is that I would expect the extrapolation (red curve) to follow the down slope of the training curve (black curve). Instead, what we see is an almost identical curve, but translated to the right of the training curve. I would appreciate any insights on why this is happening, or any comments suggesting there is one or more flaws on my logic. Lastly, it would be interesting to hear thoughts on how to achieve this "extension" of the training curve on to the extrapolation curve.

            def create_model():
                model = Sequential()
                model.add(Dense(200, input_dim=len(X_train.columns))) 
                model.add(Activation('relu'))
                model.add(Dropout(0.1))
    
                model.add(Dense(200))
                model.add(Activation('relu'))
                model.add(Dropout(0.1))
    
                model.add(Dense(200))
                model.add(Activation('relu'))
                model.add(Dropout(0.1))
                
                model.add(Dense(200))
                model.add(Activation('relu'))
                model.add(Dropout(0.1))
                
                model.add(Dense(1, activation='linear'))
                # compile the keras model
                model.compile(loss='mean_absolute_error', optimizer=tf.keras.optimizers.Adam(0.001), metrics=['mean_squared_error','mean_absolute_error'])
                return model
                    
            model_rf = Pipeline([
                ('scaler', StandardScaler()),
                ('estimator', KerasRegressor(model=create_model, epochs=200, batch_size= 1024, verbose=1))

 A: If a neural net is built with ReLu units, then its asymptotic behaviour is necessarily linear. No training can change this.
More generally, no machine learning with a finite training set can train asymptotic behaviour. So extrapolations always reflect a priori assumptions, not training.
A: After some weeks of testing, I think I finally figured out the solution for the issue above. And it is rather basic.
First, it's good to recall the two datasets used: the historical one, which was used for training and validation, and the future one, for tests purpose. Being the datasets timeseries, they were detrended. However, they were corrected by their mean values, as it is the standard approach in the field. Again, it is a basic thing that I did not realise before, but this leads to a significant bias between training data and tests data.
The solution was to adjust the detrending of the future dataset to have a mean value equal to the historical dataset. This way, with the mean values equal, the extrapolation became smooth and more similar to what I would have expected in the first place. Figure below illustrates the behaviour.

