In a set of lecture notes that I stumbled upon online, the author discusses a nonlinear regression model, which is linear in some parameters, like this $$ y = \theta_1 + \theta_2\exp \left( {\theta_3x} \right) + \varepsilon $$ and suggests to first perform a nonlinear regression to estimate $\theta_3$, and then use OLS to estimate the other two parameters. I am not very experienced in regression modeling, so please forgive me if I ask a very simple question, but I don't understand how you can do this consistently without also estimating the other 2 parameters? Apparently, there is some trick involved, which I don't understand...

Another question is what happens to the ability to do inferences based on parameter estimates to the complete model if you do the regression in two stages like that?

Many thanks!

  • 2
    $\begingroup$ You pick values for $\theta_1$ and $\theta_2$. Then minimise with respect to $\theta_3$. You get the value $\theta_3^{(1)}$. Having it you now use OLS to estimate $\theta_1^{(1)}$ and $\theta_2^{(1)}$. You got your first iteration. Then iterate until convergence. $\endgroup$ – mpiktas Apr 13 '12 at 13:37
  • $\begingroup$ Ok, thanks, but what do you gain from iterating like that compared to performing a full-blown NLS on the complete model? You essentially perform lots of OLS regressions iteratively if you run NLS using a Gauss-Newton method anyway, right? $\endgroup$ – neo123 Apr 13 '12 at 14:03
  • $\begingroup$ You do not perform lots of OLS regressions if you run NLS. The point of the procedure is that it practice it might work better than straight NLS. The NLS is guaranteed to work if your starting values are close to the optimal ones. So the outlined procedure can be thought of a way to get better starting values. Theory is nice, but in practice sometimes is very hard to get convergence. Any trick then helps. $\endgroup$ – mpiktas Apr 13 '12 at 18:54
  • $\begingroup$ Improved convergence is of course desirable. Thanks for that answer! I may be wrong, but I think you may, in principle, use OLS estimates for the so called Gauss-Newton regressions which are used for computing the step size at each iteration in a Gauss-Newton method. $\endgroup$ – neo123 Apr 13 '12 at 21:53
  • $\begingroup$ @mpiktas What would be the advantage of doing it this way instead of picking a value for $\theta_3$ and then optimizing $\theta_1$ and $\theta_2$ via OLS? $\endgroup$ – Dave Jan 9 '20 at 21:59

OLS for the model $y = \theta_1 + \theta_2 u + \varepsilon$ can be understood as a formula to convert an array of data $(u_i, y_i)$ into least squares (LS) parameter estimates $(\hat{\theta_1}, \hat{\theta_2})$. Associated with any parameter estimates (not necessarily the least squares ones) is a sum of squares of residuals. Let's call this $\text{SS}(\hat{\theta_1}, \hat{\theta_2})$.

Now, given data $((x_i,y_i),i=1,\ldots,n)$, fix a value of $\theta_3$ and let $u_i = \exp(\theta_3 x_i)$. The composite function given by

$$\lambda: \theta_3 \rightarrow ((u_i,y_i))\xrightarrow{LS} (\hat{\theta_1}, \hat{\theta_2}) \rightarrow \text{SS}(\hat{\theta_1}, \hat{\theta_2})$$

is the residual sum of squares when the LS estimates $(\hat{\theta_1}, \hat{\theta_2})$ are computed with the third parameter set to $\theta_3$. The least squares solution $\hat{\theta_3}$ is obtained by minimizing $\lambda(\theta_3)$. (The minimization can be done in any way that is appropriate and efficient; it need not even be iterative when direct solutions are available.) Once that is obtained, we take $(\hat{\theta_1}, \hat{\theta_2})$ to be the OLS estimates associated with this value of $\theta_3$. This is a considerable simplification because the minimization is a one-variable problem, not a three-variable problem.

  • $\begingroup$ Thank you for that clarification! I do see that it becomes a considerable simplification now. $\endgroup$ – neo123 Apr 13 '12 at 15:18
  • $\begingroup$ It is not immediately clear to me what happens with $\hat{\theta_3}$ if we are dealing with heteroskedastic error terms, though. If we use GLS for estimating $\hat{\theta_1}$ and $\hat{\theta_2}$, that might be fine I guess, but will $\hat{\theta_3}$ be efficient? $\endgroup$ – neo123 Apr 13 '12 at 15:46
  • 1
    $\begingroup$ There's no change. In GLS the objective function ("SS") becomes a little different, but this approach of factoring the optimization into two parts still obtains values of $\hat{\theta_i}$ that minimize the objective. After all, the values of $\hat{\theta_1}$ and $\hat{\theta_2}$ are correct because they are obtained by GLS itself, and if the value of $\hat{\theta_3}$ were wrong, then it would not minimize $\lambda$. $\endgroup$ – whuber Apr 13 '12 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.