7
$\begingroup$

In a set of lecture notes that I stumbled upon online, the author discusses a nonlinear regression model, which is linear in some parameters, like this $$ y = \theta_1 + \theta_2\exp \left( {\theta_3x} \right) + \varepsilon $$ and suggests to first perform a nonlinear regression to estimate $\theta_3$, and then use OLS to estimate the other two parameters. I am not very experienced in regression modeling, so please forgive me if I ask a very simple question, but I don't understand how you can do this consistently without also estimating the other 2 parameters? Apparently, there is some trick involved, which I don't understand...

Another question is what happens to the ability to do inferences based on parameter estimates to the complete model if you do the regression in two stages like that?

Many thanks!

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ You pick values for $\theta_1$ and $\theta_2$. Then minimise with respect to $\theta_3$. You get the value $\theta_3^{(1)}$. Having it you now use OLS to estimate $\theta_1^{(1)}$ and $\theta_2^{(1)}$. You got your first iteration. Then iterate until convergence. $\endgroup$
    – mpiktas
    Commented Apr 13, 2012 at 13:37
  • $\begingroup$ Ok, thanks, but what do you gain from iterating like that compared to performing a full-blown NLS on the complete model? You essentially perform lots of OLS regressions iteratively if you run NLS using a Gauss-Newton method anyway, right? $\endgroup$
    – neo123
    Commented Apr 13, 2012 at 14:03
  • $\begingroup$ You do not perform lots of OLS regressions if you run NLS. The point of the procedure is that it practice it might work better than straight NLS. The NLS is guaranteed to work if your starting values are close to the optimal ones. So the outlined procedure can be thought of a way to get better starting values. Theory is nice, but in practice sometimes is very hard to get convergence. Any trick then helps. $\endgroup$
    – mpiktas
    Commented Apr 13, 2012 at 18:54
  • $\begingroup$ Improved convergence is of course desirable. Thanks for that answer! I may be wrong, but I think you may, in principle, use OLS estimates for the so called Gauss-Newton regressions which are used for computing the step size at each iteration in a Gauss-Newton method. $\endgroup$
    – neo123
    Commented Apr 13, 2012 at 21:53
  • $\begingroup$ @mpiktas What would be the advantage of doing it this way instead of picking a value for $\theta_3$ and then optimizing $\theta_1$ and $\theta_2$ via OLS? $\endgroup$
    – Dave
    Commented Jan 9, 2020 at 21:59

1 Answer 1

Reset to default
7
$\begingroup$

OLS for the model $y = \theta_1 + \theta_2 u + \varepsilon$ can be understood as a formula to convert an array of data $(u_i, y_i)$ into least squares (LS) parameter estimates $(\hat{\theta_1}, \hat{\theta_2})$. Associated with any parameter estimates (not necessarily the least squares ones) is a sum of squares of residuals. Let's call this $\text{SS}(\hat{\theta_1}, \hat{\theta_2})$.

Now, given data $((x_i,y_i),i=1,\ldots,n)$, fix a value of $\theta_3$ and let $u_i = \exp(\theta_3 x_i)$. The composite function given by

$$\lambda: \theta_3 \rightarrow ((u_i,y_i))\xrightarrow{LS} (\hat{\theta_1}, \hat{\theta_2}) \rightarrow \text{SS}(\hat{\theta_1}, \hat{\theta_2})$$

is the residual sum of squares when the LS estimates $(\hat{\theta_1}, \hat{\theta_2})$ are computed with the third parameter set to $\theta_3$. The least squares solution $\hat{\theta_3}$ is obtained by minimizing $\lambda(\theta_3)$. (The minimization can be done in any way that is appropriate and efficient; it need not even be iterative when direct solutions are available.) Once that is obtained, we take $(\hat{\theta_1}, \hat{\theta_2})$ to be the OLS estimates associated with this value of $\theta_3$. This is a considerable simplification because the minimization is a one-variable problem, not a three-variable problem.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for that clarification! I do see that it becomes a considerable simplification now. $\endgroup$
    – neo123
    Commented Apr 13, 2012 at 15:18
  • $\begingroup$ It is not immediately clear to me what happens with $\hat{\theta_3}$ if we are dealing with heteroskedastic error terms, though. If we use GLS for estimating $\hat{\theta_1}$ and $\hat{\theta_2}$, that might be fine I guess, but will $\hat{\theta_3}$ be efficient? $\endgroup$
    – neo123
    Commented Apr 13, 2012 at 15:46
  • 1
    $\begingroup$ There's no change. In GLS the objective function ("SS") becomes a little different, but this approach of factoring the optimization into two parts still obtains values of $\hat{\theta_i}$ that minimize the objective. After all, the values of $\hat{\theta_1}$ and $\hat{\theta_2}$ are correct because they are obtained by GLS itself, and if the value of $\hat{\theta_3}$ were wrong, then it would not minimize $\lambda$. $\endgroup$
    – whuber ♦
    Commented Apr 13, 2012 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.