3
$\begingroup$

I read somewhere,

In a regression problem, if the relationship between each predictor variable and the criterion variable is nonlinear, then the predictions may systematically overestimate the actual values for one range of values on a predictor variable and underestimate them for another.

Does anyone have an explanation to this behavior?

$\endgroup$
6
  • 2
    $\begingroup$ nonlinearity is the explanation, assuming you used a linear regression problem. Just draw some non-linear curve approximating some straight line. $\endgroup$ Commented Nov 3, 2015 at 18:39
  • $\begingroup$ is there a way to customize this range of values? $\endgroup$ Commented Nov 3, 2015 at 19:19
  • $\begingroup$ Why do you want your algorithm to predict some wrong value? $\endgroup$ Commented Nov 3, 2015 at 19:22
  • 1
    $\begingroup$ consider, I have a dataset that it's target values range from -1 to 1. For test cases with target value near 1, I want to get predicted value larger than it's real value in test set. For example if the real value in a test case is 0.9, I want from my regression algorithm to predict it 0.95. And for test cases with target value near -1, I want to get predicted value smaller than it's real value in test set. For example if the real value in a test case is -0.9, I want to my regression algorithm to predict it -0.95. Is it possible? $\endgroup$ Commented Nov 3, 2015 at 19:48
  • 1
    $\begingroup$ The question seems perfectly clear to me. I don't see that the OP wants to do this; they just seem to want to understand it, which I think is perfectly reasonable. $\endgroup$ Commented Nov 4, 2015 at 0:49

1 Answer 1

4
$\begingroup$

Here is a simple example (coded in R). Hopefully the image is sufficiently obvious to explain how a nonlinear relationship (not model), when fit with a linear relationship, yields regions with predicted values that are systematically overestimated and underestimated.

set.seed(7439)                # this makes the example exactly reproducible
x = runif(50, min=0, max=15)  # X is uniformly distributed from 0 to 15
 # this is the true data generating process: 
y = 3.7 - 2.5*x + 0.56*x^2 - 0.028*x^3 + rnorm(50, mean=0, sd=.3)
model = lm(y~x)               # here I fit a model with a linear relationship

windows()
  plot(x, y)                       # this plots the data
  abline(coef(model), col="gray")  # this plots the model's predicted values

enter image description here

$\endgroup$
6
  • $\begingroup$ thanks, but what happened when we have nonlinear relationship and also model it with a nonlinear algorithm? $\endgroup$ Commented Nov 4, 2015 at 5:11
  • 1
    $\begingroup$ If the model's functional from is appropriate, you should be fine. $\endgroup$ Commented Nov 4, 2015 at 5:29
  • $\begingroup$ thanks, but what happened when we have nonlinear relationship and also model it with a nonlinear algorithm? I saw overestimated and underestimated predictions in low and high values of targets respectively when using nonlinear model too. $\endgroup$ Commented Nov 4, 2015 at 5:32
  • $\begingroup$ This was just answered by @gung. If you pick the correct functional form of the nonlinear model, you should be fine. There shouldn't be a systematic underestimate or overestimate. $\endgroup$ Commented Nov 4, 2015 at 5:52
  • $\begingroup$ We can go a bit further to investigate the goodness of fit. summary(model) ... Multiple R-squared: 0.04939, Adjusted R-squared: 0.02959 F-statistic: 2.494 on 1 and 48 DF, p-value: 0.1208 You can observe the R-squared. It's too low. 2% indicates that the model explains almost none of the variability of the response data around its mean. $\endgroup$ Commented Apr 20, 2020 at 4:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.