How do I examine biasedness in nonlinear regression?

Question

I have the nonlinear regression model $$y_i=\beta_0+\beta_1 x_{1i}+e^{\beta_2 x_{2i}}+u_i,\quad i=1,2,\ldots,n,$$ and the least squares assumptions are satisfied (see below). Let $\beta=(\beta_0,\beta_1,\beta_2)$.

Given this model, I want to examine the bias of the nonlinear least squares estimates of $\beta$. This estimate is given by the global solution to the minimization problem $$\min_{\beta_0,\beta_1,\beta_2}\sum_i(y_i-\beta_0-\beta_1 x_{1i}-e^{\beta_2 x_{2i}})^2.$$ The solution may be denoted by $\hat{\beta}=(\hat{\beta}_0,\hat{\beta}_1,\hat{\beta}_2)$.

(One may note that the first order conditions to the minimization problem are

\begin{cases} \sum_i(y_i-\beta_0-\beta_1 x_{1i}-e^{\beta_2 x_{2i}})&=0;\\ \sum_i(y_i-\beta_0-\beta_1 x_{1i}-e^{\beta_2 x_{2i}})x_{1i}&=0;\\ \sum_i(y_i-\beta_0-\beta_1 x_{1i}-e^{\beta_2 x_{2i}})e^{\beta_2 x_{2i}}&=0; \end{cases}

and add to this that I do not think we can find a analytic solution to this equation system. Thus, we may have to use computer algorithms to solve the system above.)

Now, I have two questions regarding $\hat{\beta}$.

A. Firstly, how do I examine the bias $E(\hat{\beta}-\beta)$? (Expectation is taken componentwise.)

B. Secondly, is the bias equal to zero for the intercept and the coefficient on $x_{1i}$? I.e., is it true that $$E(\hat{\beta}-\beta)=(0,0,E(\hat{\beta}_2-\beta_2))?$$ For the first question, A, I am looking for a method of investigation; it may involve using e.g. MATLAB or Mathematica. For the second question, B, I am looking for a proof or a counterexample. (Question B may be viewed as a subquestion to A.)

I am interested in this for it is often said that nonlinear regression introduce bias into the statistics. References for this statement are several, e.g., the Wikipedia article in English on nonlinear regression; page 51 in Nonlinear Regression, 2005, by G. A. F. Seber, C. J. Wild; Bias in Nonlinear Regression, Cook et al., Biometrika (1986), 73, 3, pp. 615-23; Bias in Nonlinear Estimation, M. J. Box, Journal of the Royal Statistical Society. Series B (Methodological) (1971), 33, 2, pp. 171-201; and the blog post Some Properties of Non-linear Least Squares by Dave Giles.

Least squares assumptions.

1. For all $i$, the error term $u_i$ has conditional mean zero given $x_{1i}$ and $x_{2i}$, i.e. $E(u_i|x_{1i},x_{2i})=0$.
2. $(y_i,x_{1i},x_{2i})$, $i=1,2,\ldots,n$, are i.i.d. draws from their joint distribution.
3. Large outliers are unlikely, meaning that for each $i$, $y_i$, $x_{1i}$ and $x_{2i}$ have nonzero finite fourth moments.
4. There is no perfect multicollinearity.

Answer 1

If you define $$z\equiv{y}-e^{\beta_2 x_2}=\beta_0+\beta_1 x_1+u$$ then given $\beta_2$, you can estimate $\beta_0$ and $\beta_1$ via OLS. This trick is useful computationally to turn your problem into a scalar nonlinear optimization over $\beta_2$.

As OLS is unbiased, any bias in the nonlinear parameter will be inherited by the linear parameters. That is, if $\beta_2$ is unbiased, then all parameters will be unbiased. But if not, then the linear parameters must also be biased. So in general the answer to your question B is no.

For your question A, numerical experiments would be a reasonable start. (Here the regime of interest would be where the range of $\beta_2x_2$ is "not too small", else the nonlinearity will be "linearized out".)

Note also that your references give some approximate formulas for estimating the bias.

---

Clarification: the optimization objective function (error) is identical when using the partitioned approach, i.e. $$E\big[\beta_{0:2}\big]=\tfrac{1}{2}\Big\|z\big[\beta_2\big]-\hat{y}_\mathrm{lin}\big[\beta_{0:1}\big]\Big\|^2$$ The "separable" approach works because for the linear sub-problem, OLS gives the global optimum (for a given $\beta_2$).

Answer 2

In order to be able to talk about whether $\hat{\beta}$ is biased, we need to have some concept of a true $\beta$. That is, we need to specify a data-generating process. So in order to make this problem tractable, let's assume that

$y_i = \beta_0 + \beta_1 x_{i1} + e^{\beta_2 x_{i2}} + u_i$

and

$u_i \sim N(0, \sigma^2)$.

Your estimator is asymptotically unbiased because it has the same solution as a maximum likelihood problem, and MLE is asymptotically unbiased.

Here's the math: \begin{align} f(y_i | x_i; b, \sigma^2) = \frac{1}{\sqrt{2 \pi \sigma^2}} e^{-(y_i - b_0 - b_1 x_{i1} - e^{b_2 x_{i2}})^2 / 2 \sigma^2} \\ \end{align} $ \log(LL(b, \sigma | y_1, \dots y_n, x_1, \dots x_n)) = -\frac{n}{2} \log(\sigma^2) - \sum_i (y_i - b_0 - b_1 x_{i1} - e^{b_2 x_{i2}})^2 / (2 \sigma^2) $

You can check that the log-likelihood has the same first-order conditions for $b_0$, $b_1$, and $b_2$ as the least-squares problem you set up, so it has the same answer. (Solving this as an MLE problem also requires estimating $\sigma^2$, but the estimate of $\sigma^2$ doesn't affect the estimates of $\beta$.)

So it's asymptotically unbiased, but what about in finite samples? I'm not sure how to prove that. Here's a sketchy "proof by Matlab", where the first three components of "params" correspond to $\beta$ and the last corresponds to $\sigma^2$::

b = [1, 1, 1]; sigma_squared = 1; n = 100; nIters = 10000; rng(10317); ll = @(y, x, b, n) sum((y - b(1) - b(2)*x(:, 1) - exp(b(3) * x(:, 2))).^2) / (2 * b(4)) + .5 * n* log(b(4)); params = zeros(niters, 4); options = optimoptions(@fminunc, 'Algorithm', 'quasi-newton') x = normrnd(0, 1, n(n_obs), 2); mean_ = b(1) + x(:, 1) * b(2) + exp(x(:, 2) * b(3)); for ii = 1:niters y = mean_ + normrnd(0, sigma_squared, n(n_obs), 1); params(ii, :) = fminunc(@(b) ll(y, x, b, n(n_obs)), [1, 1, 1, 1], options); end E_b_hat = mean(params); b_se = std(params) / (nIters - 1); bias = E_b_hat - b;

I get a near-zero and insignificant bias for $\hat{\beta}$. However, this only "proves" that $\hat{\beta}$ is unbiased for a particular value of $\beta$ and distribution of $u$ and $x$. This estimator might not be finite-sample unbiased in general.

(A general tip that doesn't apply here: When it's easy to prove biasedness in nonlinear least squares, it's often through invoking Jensen's Inequality. See this reference.)