0
$\begingroup$

I use two Machine Learning Algorithms to learn how my target variable [0, 8] is affected by four features, each within a scale of [1, 10]. I am using scikit-learn to do this task for me.

from sklearn.ensemble import RandomForestRegressor

X_train = np.array([[9.0, 8.0, 2.0, 9.0], [7.0, 9.0, 3.0, 8.0], [1.0, 2.0, 8.0, 3.0]])
y_train = np.array([8.0, 8.1, 2.2])

X_test = np.array([[10.0, 9.0, 1.8, 8.5]])

rforest = RandomForestRegressor().fit(X_train, y_train)
print("Random Forest predicts {:3.2f}".format(rforest.predict(X_test)[0]))

As you can see, I chose my X_train values such that high values for index 0, 1 and 3 and low value for index 2 result in a high value for y_train.

Now comes a new sample which the algorithm has not yet seen (X_test). Note how the first feature is 10.0 and thus higher than any other value the algorithm had learned - though still in the valid range. Despite that fact, the result is:

Random Forest predicts 8.07

If I do the same with the standard settings of an MLPR, I get the following:

from sklearn.neural_network import MLPRegressor

# [...]

mlpr = MLPRegressor().fit(X_train, y_train)
print("Neural Network (MLPR) predicts {:3.2f}".format(mlpr.predict(X_test)[0]))

Neural Network (MLPR) predicts 8.61

The prediction of the NN is outside of the range it learned (y_train). It learned that an exaggeration of any of the features results in an exaggeration of the predicted variable.

This may be of interest for many studies, but in my case I am rather restricted in the range of the predictions. A prominent example would be how a NN should predict concentrations of a certain chemical substance, but returns negative concentrations based on the features it learned.

My problem is that I cannot specify the valid range, only that I would like predictions to be within the range it originally learned.

How is such an algorithm called and is there a way to restrict NN or SVRs to the range in which they were trained (just like the random forest does)?

$\endgroup$
1
$\begingroup$

If you were using linear regression, you wouldn't be surprised that an out-of-range input corresponded to an out-of-range output. :) It's the same for a neural network, whose function is more complex, but still unbounded depending on your final activation function.

If you want to constrain the output range, you can clip the network's output values.

$\endgroup$
  • $\begingroup$ Thanks for your answer! Would you say "unbounded" is a valid term for this phenomenon? I need to justify why I used a certain algorithm. I train the MLPR with simulated data, but use it on real data which has many features that are outside of what the algorithm learns. Clipping unfortunately is not sufficient to me, as there are also unrealistic high values. Maybe I will just stick with Random Forests then. Anyway, I do see the point and why this happens. Just thought there might be a way to avoid extrapolation... $\endgroup$ – offeltoffel Dec 5 '18 at 6:32
  • 1
    $\begingroup$ In general, people clip the outputs. One common way is to use sigmoid to softly clip to [0, 1]. Similarly, softmax is used with categorical output to force the logics to sum to 1. $\endgroup$ – alltom Dec 5 '18 at 6:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.