How is it called when a MLR algorithm predicts a value beyond the range of the training data set and is there a way to avoid this for Neural Networks?

Question

I use two Machine Learning Algorithms to learn how my target variable [0, 8] is affected by four features, each within a scale of [1, 10]. I am using scikit-learn to do this task for me.

from sklearn.ensemble import RandomForestRegressor

X_train = np.array([[9.0, 8.0, 2.0, 9.0], [7.0, 9.0, 3.0, 8.0], [1.0, 2.0, 8.0, 3.0]])
y_train = np.array([8.0, 8.1, 2.2])

X_test = np.array([[10.0, 9.0, 1.8, 8.5]])

rforest = RandomForestRegressor().fit(X_train, y_train)
print("Random Forest predicts {:3.2f}".format(rforest.predict(X_test)[0]))

As you can see, I chose my X_train values such that high values for index 0, 1 and 3 and low value for index 2 result in a high value for y_train.

Now comes a new sample which the algorithm has not yet seen (X_test). Note how the first feature is 10.0 and thus higher than any other value the algorithm had learned - though still in the valid range. Despite that fact, the result is:

Random Forest predicts 8.07

If I do the same with the standard settings of an MLPR, I get the following:

from sklearn.neural_network import MLPRegressor

# [...]

mlpr = MLPRegressor().fit(X_train, y_train)
print("Neural Network (MLPR) predicts {:3.2f}".format(mlpr.predict(X_test)[0]))

Neural Network (MLPR) predicts 8.61

The prediction of the NN is outside of the range it learned (y_train). It learned that an exaggeration of any of the features results in an exaggeration of the predicted variable.

This may be of interest for many studies, but in my case I am rather restricted in the range of the predictions. A prominent example would be how a NN should predict concentrations of a certain chemical substance, but returns negative concentrations based on the features it learned.

My problem is that I cannot specify the valid range, only that I would like predictions to be within the range it originally learned.

How is such an algorithm called and is there a way to restrict NN or SVRs to the range in which they were trained (just like the random forest does)?

Answer 1

If you were using linear regression, you wouldn't be surprised that an out-of-range input corresponded to an out-of-range output. :) It's the same for a neural network, whose function is more complex, but still unbounded depending on your final activation function.

If you want to constrain the output range, you can clip the network's output values.