Suppose I'm attempting to use linear regression but instead of

$$y_i = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3 + \beta_4 x_4 + \varepsilon_i,$$

for scientific reasons, I know that $β_2=\alpha\beta_1$ and $β_4=\alpha\beta_3$. So my formula is actually:

$$y_i = \beta_0 + \beta_1 x_1 + \alpha\beta_1 x_2 + \beta_3 x_3 + \alpha\beta_3 x_4 + \varepsilon_i$$

Note that the variable $\alpha$ is the same in both equations but unknown. How should I tackle something like this?

  • $\begingroup$ Hopefully someone else can give a good answer. But if they don't you could probably do a brute-force optimization with a numerical optimizer, e.g. optim() in R. $\endgroup$
    – wzbillings
    Dec 13, 2022 at 2:12
  • $\begingroup$ This is straightforward to do by minimizing the usual least squares objective $||y - X\beta||^2$ subject to the constraint $\beta^\prime Q \beta = 0$ where $\beta^\prime Q \beta = \beta_2\beta_3 - \beta_1\beta_4.$ Introduce a Lagrange multiplier. $\endgroup$
    – whuber
    Dec 13, 2022 at 14:54
  • $\begingroup$ @whuber is there an R package that allows you to work with Lagrange multipliers? $\endgroup$
    – John Henry
    Dec 13, 2022 at 16:17
  • $\begingroup$ I am sure there are many, because this is a standard quadratic program with a quadratic constraint. It's a little special, though, because $Q$ is not a definite quadratic form, so it might take some research to find a package that works correctly in this case. Thus, it's probably better to work through the mathematics, which ought to reduce the problem to one solvable with standard linear algebra techniques. $\endgroup$
    – whuber
    Dec 13, 2022 at 16:26

3 Answers 3


Since $\alpha$ is unknown, this reduces to a nonlinear regression with the model form:

$$y_i = \beta_0 + \beta_1 [x_1 + \alpha x_2] + \beta_2 [x_3 + \alpha x_4] + \varepsilon_i.$$

You can implement nonlinear regression using the nls function in R. For a nonlinear model you will need to specify a starting point for your parameters in order to implement the iterative fitting-method. A reasonable starting point for the parameters in this case could be obtained by fitting the unconstrained linear model (your first model) and then taking the starting values for the parameters to be:

$$\begin{align} \beta_0^* &= \hat{\beta}_0, \\[12pt] \beta_1^* &= \hat{\beta}_1, \\[12pt] \beta_2^* &= \hat{\beta}_3, \\[6pt] \alpha^* &= \frac{\hat{\beta}_2+\hat{\beta}_4}{\hat{\beta}_1+\hat{\beta}_3}. \\[6pt] \end{align}$$

(The values on the right-hand-side in these equations would be estimates from your first model.)


I am thinking of two different ways with different types of analysis.

A) you simply run the regression as if they are no relationship. In R, you can use model = lm(y ~ x1 + x2 + x3 + x4). Once you get the coefficient values from summary(model) or coef(model), you use linear equations, where: $$ \hat{\beta_2}=\hat{\alpha}\hat{\beta_1} $$ OR $$ \hat{\beta_4}=\hat{\alpha}\hat{\beta_3} $$

The only concern I have is that both values of $\hat{\alpha}$ is no similar.

B) Imposing conditions as you stated. I am not sure how to run this in R but I think Solver in Excel can help you.

My question to you: How does the coefficient has relationship with one another? I usually find the case where the "so-called" independent variables are actually not independent, meaning $x_1=f(x_2)$ and $x_3=f(x_4)$


This is essentially a non-linear least squares problem, and the R function nls may be of interest to you. Rather than describe the workings of the function though, let me briefly outline the relevant statistical theory.

The basic idea behind non-linear least squares is as follows. You have some regression function $f(X;\theta)$ which takes predictors $X$ and parameters $\theta$ and predicts the value of the outcome $y$. In particular, the assumption is that $\mathbb E[y|X] = f(X;\theta)$ for some $\theta$ in your case, $\theta = (\beta_1,\beta_3,\alpha)$ and $f(x_1,x_2,x_3,x_4;\theta) = \beta_1(x_1+\alpha x_2) + \beta_3(x_3 + \alpha x_4)$. The fact that $f(X;\theta)$ is the conditional expectation of $y$ given $X$ implies that in population, we have that the true value of the parameter, $\theta_0$ satisfies $$\theta_0 = \mathrm{argmin}_\theta\, \mathbb E[(y - f(X;\theta))^2]$$ The fact that $\theta_0$ solves this optimization problem immediately implies an estimation strategy: simply choose $\hat\theta$ to be the value of the parameter that minimizes the in-sample analogue of the above objective function, that is $$\hat\theta = \mathrm{argmin}_\theta\, \frac1n\sum_{i=1}^n (y_i - f(X_i;\theta))^2$$ This is the reason for @wzbillings's suggestion above to use the R optim function.

If you are interested in how, given this estimate, you should compute standard errors, you can consult previous posts, such as this one.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.