Skip to main content
Cross Validated

Return to Answer

added 1286 characters in body
Source Link
Acccumulation
Acccumulation
  • 5.3k
  • 9
  • 14

If[Edited as I misread the original question]

The way a linear regressionRandom Forest works, simplified, is getting consistent errorsthat the computer tries to sort the observations into buckets, then calculates the average value within each bucket, and uses that means your data has a consistent non-linear component to itas the score for everything in that bucket. In this caseThis is done repeatedly, your data has somewhatgetting different bucketing rules, and then the final score for an observation is the average of a sigmoid shape tothe scores for all the buckets it was in. Try taking $f(y) = \ln(1-(y-a)/b)$ for various values

One issue with Random Forest is that it can "top out": a set of $a$ andobservations can be sorted into the "highest bucket" $b$(or the "lowest bucket"), and see whetherfrom there, no further distinction is made within the linear regressionset. If the same set is topping out for $f(y)$ versuseach bucketing rule, then you're not going to be getting a different score for them.

Something along these lines appears to be happening with your predictions. Your model isn't predicting less than $x$ is better-1, or much more than 1, which suggests that the set all observations with a true value greater than 1 is topping out, and vice versa for $y$ versusless than $x$-1. Also

Now, it looks like most of your data is centralalso seems to be much more sparse for those data points, so it's natural for your model to prioritize gettingis probably prioritizing the more central data correctones. If you're more concerned aboutFurthermore, you may not have enough datapoints, regardless of what you do, to make predictions for those observations. Or the explanatory variable may just not be explanatory anymore at the extremes (for instance, then you'llage is a good predictor of height for schoolchildren, but not so much for adults; if you have to weight them accordinglythat sort of relationship, there might not be anything you can do). But things you can try are:

  1. Create separate models for the extremes
  2. Look at your extremes and try to find distinguishing characteristics
  3. Generate new features, such as doing a linear regression on the original features, and feeding the output of that into the Random Forest model as an input.

If a linear regression is getting consistent errors, then that means your data has a consistent non-linear component to it. In this case, your data has somewhat of a sigmoid shape to it. Try taking $f(y) = \ln(1-(y-a)/b)$ for various values of $a$ and $b$ and see whether the linear regression for $f(y)$ versus $x$ is better than that for $y$ versus $x$. Also, it looks like most of your data is central, so it's natural for your model to prioritize getting the central data correct. If you're more concerned about the extremes, then you'll have to weight them accordingly.

[Edited as I misread the original question]

The way a Random Forest works, simplified, is that the computer tries to sort the observations into buckets, then calculates the average value within each bucket, and uses that as the score for everything in that bucket. This is done repeatedly, getting different bucketing rules, and then the final score for an observation is the average of the scores for all the buckets it was in.

One issue with Random Forest is that it can "top out": a set of observations can be sorted into the "highest bucket" (or the "lowest bucket"), and from there, no further distinction is made within the set. If the same set is topping out for each bucketing rule, then you're not going to be getting a different score for them.

Something along these lines appears to be happening with your predictions. Your model isn't predicting less than -1, or much more than 1, which suggests that the set all observations with a true value greater than 1 is topping out, and vice versa for less than -1.

Now, your data also seems to be much more sparse for those data points, so your model is probably prioritizing the more central ones. Furthermore, you may not have enough datapoints, regardless of what you do, to make predictions for those observations. Or the explanatory variable may just not be explanatory anymore at the extremes (for instance, age is a good predictor of height for schoolchildren, but not so much for adults; if you have that sort of relationship, there might not be anything you can do). But things you can try are:

  1. Create separate models for the extremes
  2. Look at your extremes and try to find distinguishing characteristics
  3. Generate new features, such as doing a linear regression on the original features, and feeding the output of that into the Random Forest model as an input.
Source Link
Acccumulation
Acccumulation
  • 5.3k
  • 9
  • 14

If a linear regression is getting consistent errors, then that means your data has a consistent non-linear component to it. In this case, your data has somewhat of a sigmoid shape to it. Try taking $f(y) = \ln(1-(y-a)/b)$ for various values of $a$ and $b$ and see whether the linear regression for $f(y)$ versus $x$ is better than that for $y$ versus $x$. Also, it looks like most of your data is central, so it's natural for your model to prioritize getting the central data correct. If you're more concerned about the extremes, then you'll have to weight them accordingly.