Given a non-linear regression model for cross-section data

$$y_i = f(x_i,\theta_0) + \epsilon_i,$$

where it is assumed that $\mathbb E[y_i\lvert x_i] = f(x_i,\theta_0)$, I understand that it is a standard approach to estimate the conditional variance of the errors $Var(\epsilon_i \lvert x_i)$ as a function of the covariates $\sigma^2(x_i,\lambda)$ and that the non-linear least squares estimator $\hat \theta$ remains consistent even though the variance function may be misspecified. However I am not really sure I understand why the misspecification of the condtional variance do not imply inconsistency of $\hat \theta$?


Consistency depends on how you let the $x_i$ increase in number.

If you have the following model (which is in fact a linear model):

$y_i \sim N(\mu = \theta \frac{1}{\sqrt{x_i}}, \sigma^2 = 1/x_i^2)$


$x_i \sim \text{Uniform}(0,1)$

then the variance of the estimate $\hat{\theta}$ will increase when the number of samples increases.

See the simulation below:


# function to make an estimate
samplepar <- function(n) {
  a <- 2
  x <- runif(n)
  y <- a/sqrt(x)+rnorm(n,0,0.01*a/x)
  mod <- nls(y ~ a/sqrt(x), start = list(a=2))
  out1 <- coefficients(mod)
  mod <- nls(y ~ a/sqrt(x), start = list(a=2), weights = x^2)
  out2 <- coefficients(mod)

# some settings    
n <- 10000

# perform multiple times an estimate for samples of size 5 and size 1000
small <- replicate(n,samplepar(5))
large <- replicate(n,samplepar(1000))

# compute histograms and plotting
d <- 0.01
h1 <- hist(small[1,],
           breaks = seq(min(small[1,]-d),
           plot = FALSE)
h2 <- hist(large[1,],
           breaks = seq(min(large[1,]-d),
           plot = FALSE)

plot(h1$mids,h1$counts/n/d, xlim = c(1.5,2.5),log="y",ylim=c(0.1,100),
     xlab = "estimate", ylab = "density",yaxt="n")
points(h2$mids,h2$counts/n/d, xlim = c(1.5,2.5),col=2)
legend(1.5,100,c("5 data points","1k data points"),col=c(1,2),pch=1, cex = 0.7)
title("distribution of parameter estimates \n non-weighted least squares", cex.main=1)

d <- 0.001
h1 <- hist(small[2,],
           breaks = seq(min(small[2,]-d),
           plot = FALSE)
d2 <- 0.0001
h2 <- hist(large[2,],
           breaks = seq(min(large[2,]-d2),
           plot = FALSE)

plot(h1$mids,h1$counts/n/d, xlim = c(1.95,2.05),log="y",ylim=c(1,1000),
     xlab = "estimate", ylab = "density",yaxt="n")
points(h2$mids,h2$counts/n/d2, xlim = c(1.5,2.5),col=2)
legend(1.95,1000,c("5 data points","1k data points"),col=c(1,2),pch=1, cex = 0.7)
title("distribution of parameter estimates \n weighted least squares", cex.main=1)

  • So the estimate is not stable (because the higher number of points will increase the probability to encounter high variance points when $x_i \approx 0$).

  • But if you would let the sample size increase while keeping the $x_i$ fixed (e.g. an increase of the number of samples with the same $x_i$ and variance) then the estimate should be consistent.

    Intuitive explanation: effectively you could replace the duplicate samples for a single sample but with smaller variance. Minimizing the least squares for repeated samples is the same as minimizing the least squares for the means of those repeated samples.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.