2
$\begingroup$

I'm using sklearn's RandomForestRegressor to try and model a relationship that involves three Feature variables (x1,x2,x3) and one Target variable (y1). My model appears to pass some basic validation-criteria tests, however, when I push fresh data into the model the results are way off from what I'd expect.

Building & Validating the Model

My training data looks like this (see image #1). In this visualization, each X-axis represents a different Feature and the Y-axis represents the same Target data for all three plots. This data is an array of size [658,3] prior to any test-train splitting.

Here are the important bits of the code I'm using to build the model:

X = np.vstack((x1, x2, x3))
X = np.ndarray.transpose(X)  # Input columns
y = y1  # Target data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_ratio)  # 0.2 test ratio

conversion_model = RandomForestRegressor(n_estimators=150, random_state=0)
conversion_model.fit(X_train, y_train)

Running this model through sklearn's cross_val_score function, with a cv=10, produces an average cross validation score of ~0.99. Here are some other statistics of the model when I crunch the predicted (train) vs. true (test) results.

  • RMSE = 0.005
  • Max Error = 0.017
  • R2 = 0.999

As far as I can tell, this looks like a pretty happy model.

Predicting Values Using the Model

Here's where the issues arise. I'm feeding the following dataset (see image #2) into this newly generated Random Forest model. In the image, the X-axes are the x1 and x2 features, respectively; the Y-axis is the x3 feature. Just using Image #2 as a means of sanity checking my experimentally-collected data.

It's also useful to note that Features x1 and x2 do not vary significantly between the model training data and the prediction data. Feature x3 is the main term that will shift around b/w the training set and the prediction set.

The code I'm using to capture predictions for the new Feature data is:

Xpred = np.vstack((x1_new, x2_new, x3_new))
Xpred = np.ndarray.transpose(Xpred)
ypred = conversion_model.predict(Xpred)

When I plot the resulting predictions I end up with something like this (see image #3). Obviously, there is something funky with these results. The lower-end datapoints look okay (?) but the remainder of the prediction data is plateauing and definitely not following the expected trend.

Could really use some help identifying my issue here. Thanks!


Image 1: MODEL INPUT DATA VISUALIZATION Visual of the model input data before test-train split. X-axes are the three features and Y-axis is the Target data for all three plots.

Image 2: NEW INPUT DATA, TO BE PREDICTED Visualization of the predictive input data. X-axes are the x1 and x2 features and the Y-axis is the x3 feature.

Image 3: MODEL PREDICTIONS (BAD PREDICTIONS) Predictions using the Random Forest model and the input data in Image 2.

$\endgroup$
3
  • 2
    $\begingroup$ I gather that Random Forest (and other ensemble methods) will never make predictions that are outside of the range of the original training dataset. Given that the upper bound of my prediction dataset (x3, specifically) is greater than the upper bound of my training dataset; could that explain the plateau response that I'm seeing? $\endgroup$ Jun 23 at 21:26
  • 1
    $\begingroup$ Yes, it would. The upper bound will be put into a group of values that are all less than the upper bound, that group being from the training data, and will predict on that basis. $\endgroup$
    – jbowman
    Jun 23 at 22:13
  • $\begingroup$ Thanks for the feedback @jbowman! Do you have any recommendations for a workaround such that I can artificially set/ trick/ coerce the data to predict beyond the bounds of the training set? I plan to grow my training dataset by 3-4x more, so hopefully that should help a little. But now I wonder if a different model type would be preferable $\endgroup$ Jun 23 at 22:51

1 Answer 1

1
$\begingroup$

In general, no supervised learning algorithm can safely make predictions outside the bounds of your training data. There are exceptions of course: a linear model does fine outside its domain if and only if the underlying natural system it is making predictions about is also a linear system. To put it another way: the model only knows about things it has seen before.

This limitation is one of the reasons we generally assume and require that data are "independent and identically distributed". Your data are not identically distributed with respect to the x3 feature (and potentially with respect to the target), so you're getting bad results.

As for what you can do now, here are some ideas:

  1. Find some data examples in the data domain in which you'd like to make predictions.
  2. Select a learning algorithm that behaves according to your expectations or some other information.
  3. Find a purely analytic solution (e.g. using physics or economic theory or whatever).

With option 2 you are essentially saying that you have some new information (a conceptual or physics-based model) that provides expectations for this new domain. You can use that to choose an appropriate model. For example, this figure (source) illustrates how different regressors behave outside the training domain (i.e. at the edges of these plots):

Comparison of regressors

With option 3 you're really just giving up on machine learning. It may or may not be an option for you.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.