# Non-Linear regression and variance misspecification

Given a non-linear regression model for cross-section data

$$y_i = f(x_i,\theta_0) + \epsilon_i,$$

where it is assumed that $$\mathbb E[y_i\lvert x_i] = f(x_i,\theta_0)$$, I understand that it is a standard approach to estimate the conditional variance of the errors $$Var(\epsilon_i \lvert x_i)$$ as a function of the covariates $$\sigma^2(x_i,\lambda)$$ and that the non-linear least squares estimator $$\hat \theta$$ remains consistent even though the variance function may be misspecified. However I am not really sure I understand why the misspecification of the condtional variance do not imply inconsistency of $$\hat \theta$$?

Consistency depends on how you let the $$x_i$$ increase in number.

If you have the following model (which is in fact a linear model):

$$y_i \sim N(\mu = \theta \frac{1}{\sqrt{x_i}}, \sigma^2 = 1/x_i^2)$$

with

$$x_i \sim \text{Uniform}(0,1)$$

then the variance of the estimate $$\hat{\theta}$$ will increase when the number of samples increases.

See the simulation below:

# function to make an estimate
samplepar <- function(n) {
a <- 2
x <- runif(n)
y <- a/sqrt(x)+rnorm(n,0,0.01*a/x)
mod <- nls(y ~ a/sqrt(x), start = list(a=2))
out1 <- coefficients(mod)
mod <- nls(y ~ a/sqrt(x), start = list(a=2), weights = x^2)
out2 <- coefficients(mod)
c(out1,out2)
}

# some settings
layout(matrix(1:2,1))
set.seed(1)
n <- 10000

# perform multiple times an estimate for samples of size 5 and size 1000
small <- replicate(n,samplepar(5))
large <- replicate(n,samplepar(1000))

# compute histograms and plotting
d <- 0.01
h1 <- hist(small[1,],
breaks = seq(min(small[1,]-d),
max(small[1,]+d),d),
plot = FALSE)
h2 <- hist(large[1,],
breaks = seq(min(large[1,]-d),
max(large[1,]+d),d),
plot = FALSE)

plot(h1$$mids,h1$$counts/n/d, xlim = c(1.5,2.5),log="y",ylim=c(0.1,100),
xlab = "estimate", ylab = "density",yaxt="n")
axis(2,at=c(0.1*c(2:9),1*c(2:9),10*c(2:9)),labels=rep("",24),las=2)
axis(2,at=c(0.1,1,10),las=2)
points(h2$$mids,h2$$counts/n/d, xlim = c(1.5,2.5),col=2)
legend(1.5,100,c("5 data points","1k data points"),col=c(1,2),pch=1, cex = 0.7)
title("distribution of parameter estimates \n non-weighted least squares", cex.main=1)

d <- 0.001
h1 <- hist(small[2,],
breaks = seq(min(small[2,]-d),
max(small[2,]+d),d),
plot = FALSE)
d2 <- 0.0001
h2 <- hist(large[2,],
breaks = seq(min(large[2,]-d2),
max(large[2,]+d2),d2),
plot = FALSE)

plot(h1$$mids,h1$$counts/n/d, xlim = c(1.95,2.05),log="y",ylim=c(1,1000),
xlab = "estimate", ylab = "density",yaxt="n")
axis(2,at=c(1*c(2:9),10*c(2:9),100*c(2:9)),labels=rep("",24),las=2)
axis(2,at=c(1,10,100,1000),las=2)
points(h2$$mids,h2$$counts/n/d2, xlim = c(1.5,2.5),col=2)
legend(1.95,1000,c("5 data points","1k data points"),col=c(1,2),pch=1, cex = 0.7)
title("distribution of parameter estimates \n weighted least squares", cex.main=1)


• So the estimate is not stable (because the higher number of points will increase the probability to encounter high variance points when $$x_i \approx 0$$).

• But if you would let the sample size increase while keeping the $$x_i$$ fixed (e.g. an increase of the number of samples with the same $$x_i$$ and variance) then the estimate should be consistent.

Intuitive explanation: effectively you could replace the duplicate samples for a single sample but with smaller variance. Minimizing the least squares for repeated samples is the same as minimizing the least squares for the means of those repeated samples.