2
$\begingroup$

It is well known that the (sample) mean of residuals becomes zero in an unweighted OLS fitting of a linear regression model. How about the case for the nonlinear model?

I have numerically tested whether the mean of residuals become zero in least-squares fitting of the following models for example.

y = b1 / (x + b2) + error

y = b1 * exp(b2 * x) + error

I had expected that for such nonlinear models without intercept to be estimated, the mean of residuals (predicted y - observed y) would not be generally zero. However, according to my numerical tests, where I generated the error term by using independent & zero-mean Gaussian pseudo-random numbers, the mean of residuals did not significantly deviated from zero.

Could anyone tell me whether the mean of residuals generally equals zero in unweighted least-squares fitting of nonlinear regression models?

EDIT 230728 18:40JST

I'm sorry some important conditions were missing. I'm interested in a case where y is a continuous variable and has an independent zero-mean observation error (i.e., noise) following a Gaussian distribution. In other words, the observation of y is generated by adding the true predictor of the model (with true coefficients) with an i.i.d. Gaussian error. For example, if the true model is y = 1 / (x + 2) + error where error follows standard normal distribution, the observations of y should be generated by

    y <- 1 / (x + 2) + rnorm(N, mean = 0, sd = 1)
$\endgroup$

1 Answer 1

2
$\begingroup$

NO

First, note that what you wrote only applies when there is an intercept. If your linear model lacks an intercept, then the mean of the residuals is not guaranteed to be zero.

For an example of a nonlinear model that uses the minimization of the sum of squares residuals to estimate the regression coefficients, consider the model below.

$$ \hat y_i = e^{ \hat\beta x_i } \iff \log(\hat y_i)=\hat\beta x_i $$

I can fit this to some simulated data in software, using least squares as the function to minimize to calculate the estimated $\hat\beta$.

set.seed(2023)
N <- 100
x <- rnorm(N)
y <- rbinom(N, 5, 0.5) + 1
L <- glm(y ~ 0 + x, family = gaussian(link = "log"))
mean(resid(L)) # About 2

I get a mean of the residuals of about $2$.

The family = Gaussian is what tells the software to minimize the sum of squared residuals, and link = “log” introduces nonlinearity.

EDIT

Even better than the comment might be an example with a totally continuous y with a $U(5, 6)$ distribution (both conditional and marginal distribution).

set.seed(2023)
N <- 100
x <- rnorm(N)
y <- runif(N, 5, 6)
L <- glm(y ~ 0 + x, family = gaussian(link = "log"))
mean(resid(L))   # About 4
t.test(resid(L)) # p < 2.2e-16, strong evidence against a mean of zero

EDIT 2

I'm interested in a case where y is a continuous variable and has an independent zero-mean observation error (i.e., noise) following a Gaussian distribution. In other words, the observation of y is generated by adding the true predictor of the model (with true coefficients) with an i.i.d. Gaussian error.

The examples below seems to satisfy your criteria, and the means of the residuals are not zero.

set.seed(2023)
N <- 100
x <- runif(N, 1000, 2000)
y <- log(x) + rnorm(N)
L <- glm(y ~ 0 + x, family = gaussian(link = "log"))
mean(resid(L))   # I get about 1
t.test(resid(L)) # p < 0.0001
set.seed(2023)
N <- 100
x <- runif(N, 0, 0.01)
y <- y <- 100 / (x + 2) + rnorm(N, mean = 0, sd = 0.001) # Based on the given example
L <- glm(y ~ 0 + x, family = gaussian(link = "log"))
mean(resid(L))   # I get about 29
t.test(resid(L)) # p < 2.2e-16, strong evidence against a mean of zero
$\endgroup$
9
  • $\begingroup$ Thank you for your detailed and helpful answer. I have run your GLM example script and confirmed that the mean of the residuals is not zero indeed. On the other hand, I am still not successful in making a case where y is a continuous variable and its observations are generated by y <- truey + rnorm(N, 0, sigma). $\endgroup$ Jul 27, 2023 at 5:28
  • $\begingroup$ Let me report an example non-successful case of mine. n.data <- 100 X <- runif(n = n.data, min = -3, max = 3) Y.true <- exp(0.65 * X) Y.obs <- Y.true + rnorm(n = n.data, mean = 0, sd = 1) I generated X and Y.obs in this way and then fitted the predictor exp(coeff * X) to minimize RMS residuals (optimizing coeff), then calculated the mean of residuals (MR). After 1000 such trials, i got the following statistics for MR. mean of MR: 0.00268. standard error (S.D/sqrt(1000)) of MR: 0.00268. Obviously, I cannot say the residuals are biased from this example. $\endgroup$ Jul 27, 2023 at 5:49
  • $\begingroup$ @GreatJourney I do not follow what you’re doing or how it relates to what I’ve written about residual means in nonlinear models. Could you please clarify? $\endgroup$
    – Dave
    Jul 27, 2023 at 14:43
  • $\begingroup$ I am sorry. I'd like to get a theoretical or numerical proof that the mean of residuals does not always become zero in least-squares fitting of nonlinear regression models. Your first answer indeed gave me the numerical proof. However, the nonlinear models in my mind were those with continuous objective variables (y) (I am sorry that I didn't explicitly mention it in my question). That's why, I'd like to get another example where y is continuous and the mean of residuals (MR) significantly deviates from zero. $\endgroup$ Jul 28, 2023 at 3:00
  • 1
    $\begingroup$ @GreatJourney I have provided another simulation that seems to satisfy your requirements. In the future, if you come up with a new question after the question you asked has been answered, please post a new question. $\endgroup$
    – Dave
    Jul 29, 2023 at 0:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.