Non-Linear regression and variance misspecification

Question

Given a non-linear regression model for cross-section data

$$y_i = f(x_i,\theta_0) + \epsilon_i,$$

where it is assumed that $\mathbb E[y_i\lvert x_i] = f(x_i,\theta_0)$, I understand that it is a standard approach to estimate the conditional variance of the errors $Var(\epsilon_i \lvert x_i)$ as a function of the covariates $\sigma^2(x_i,\lambda)$ and that the non-linear least squares estimator $\hat \theta$ remains consistent even though the variance function may be misspecified. However I am not really sure I understand why the misspecification of the condtional variance do not imply inconsistency of $\hat \theta$?

Answer 1

Consistency depends on how you let the $x_i$ increase in number.

If you have the following model (which is in fact a linear model):

$y_i \sim N(\mu = \theta \frac{1}{\sqrt{x_i}}, \sigma^2 = 1/x_i^2)$

with

$x_i \sim \text{Uniform}(0,1)$

then the variance of the estimate $\hat{\theta}$ will increase when the number of samples increases.

See the simulation below:

# function to make an estimate
samplepar <- function(n) {
  a <- 2
  x <- runif(n)
  y <- a/sqrt(x)+rnorm(n,0,0.01*a/x)
  mod <- nls(y ~ a/sqrt(x), start = list(a=2))
  out1 <- coefficients(mod)
  mod <- nls(y ~ a/sqrt(x), start = list(a=2), weights = x^2)
  out2 <- coefficients(mod)
  c(out1,out2)
}

# some settings    
layout(matrix(1:2,1))
set.seed(1)
n <- 10000

# perform multiple times an estimate for samples of size 5 and size 1000
small <- replicate(n,samplepar(5))
large <- replicate(n,samplepar(1000))

# compute histograms and plotting
d <- 0.01
h1 <- hist(small[1,],
           breaks = seq(min(small[1,]-d),
                            max(small[1,]+d),d),
           plot = FALSE)
h2 <- hist(large[1,],
           breaks = seq(min(large[1,]-d),
                        max(large[1,]+d),d),
           plot = FALSE)

plot(h1$mids,h1$counts/n/d, xlim = c(1.5,2.5),log="y",ylim=c(0.1,100),
     xlab = "estimate", ylab = "density",yaxt="n")
axis(2,at=c(0.1*c(2:9),1*c(2:9),10*c(2:9)),labels=rep("",24),las=2)
axis(2,at=c(0.1,1,10),las=2)
points(h2$mids,h2$counts/n/d, xlim = c(1.5,2.5),col=2)
legend(1.5,100,c("5 data points","1k data points"),col=c(1,2),pch=1, cex = 0.7)
title("distribution of parameter estimates \n non-weighted least squares", cex.main=1)


d <- 0.001
h1 <- hist(small[2,],
           breaks = seq(min(small[2,]-d),
                        max(small[2,]+d),d),
           plot = FALSE)
d2 <- 0.0001
h2 <- hist(large[2,],
           breaks = seq(min(large[2,]-d2),
                        max(large[2,]+d2),d2),
           plot = FALSE)

plot(h1$mids,h1$counts/n/d, xlim = c(1.95,2.05),log="y",ylim=c(1,1000),
     xlab = "estimate", ylab = "density",yaxt="n")
axis(2,at=c(1*c(2:9),10*c(2:9),100*c(2:9)),labels=rep("",24),las=2)
axis(2,at=c(1,10,100,1000),las=2)
points(h2$mids,h2$counts/n/d2, xlim = c(1.5,2.5),col=2)
legend(1.95,1000,c("5 data points","1k data points"),col=c(1,2),pch=1, cex = 0.7)
title("distribution of parameter estimates \n weighted least squares", cex.main=1)

---

• So the estimate is not stable (because the higher number of points will increase the probability to encounter high variance points when $x_i \approx 0$).

• But if you would let the sample size increase while keeping the $x_i$ fixed (e.g. an increase of the number of samples with the same $x_i$ and variance) then the estimate should be consistent.

  Intuitive explanation: effectively you could replace the duplicate samples for a single sample but with smaller variance. Minimizing the least squares for repeated samples is the same as minimizing the least squares for the means of those repeated samples.