# Linear regression with constrained coefficient

I have to perform a linear regression of the form $y=ax_1+bx_2+c$ where $b=(1-a)$ and I have errors both on the dependent variable y and on the predictors $x_1, x_2$. I am not sure how to handle the constraint on the second coefficient and the errors on the dependent variable. Any idea?

• Your constraint implies that you are regressing $y$ on a single variable $x_1 + x_2$ and forcing its coefficient to be $1$. That doesn't solve the problem of errors in predictors. Errors in the dependent variable are what you expect with regression. Dec 9, 2013 at 13:32
• Ok, thanks. But I am not sure how should I force the coefficient of (x1+x2) to be 1.. Can I do that with a least square fitting? Dec 9, 2013 at 13:49
• You can constrain things with least squares by reparameterizing (let y* = y-x2 and x* = x1-x2. Fit y* = ax* + c and then let b = 1-a. However, that won't deal with your errors-in-variables. In some cases at least, though, reparameterizing might work there as well. Dec 9, 2013 at 14:18

## General constrained OLS problem

Recall that the OLS problem, subject to linear constraints can be written as \begin{align} \arg\min_{\boldsymbol{\beta}}\boldsymbol{Y}'\boldsymbol{Y} - \boldsymbol{Y}'\mathbf{X}\boldsymbol{\beta} - \boldsymbol{\beta}'\mathbf{X}'\boldsymbol{Y} + \boldsymbol{\beta}'\mathbf{X}'\mathbf{X}\boldsymbol{\beta} \end{align}\\ \text{subject to }\quad \mathbf{a}\boldsymbol{\beta} = \boldsymbol{c} where in the general case, $\mathbf{a}$ is a matrix, and $\boldsymbol{c}$ is a vector.

Since the first term does not depend on $\boldsymbol{\beta}$, that we can scale by a constant without changing the solution, and that a scalar is its own transpose, we get \begin{align} \arg\min_{\boldsymbol{\beta}} - \boldsymbol{Y}'\mathbf{X}\boldsymbol{\beta} +\tfrac{1}{2} \boldsymbol{\beta}'\mathbf{X}'\mathbf{X}\boldsymbol{\beta} \end{align}\\ \text{subject to }\quad \mathbf{a}\boldsymbol{\beta} = \boldsymbol{c}

Note: I do this so that it maps neatly into the way R solves constrained quadratic programming problems.

## Specific case

In your case of three coefficients including the intercept and one constraint, \begin{align} \mathbf{a} &= [0, 1, 1] \\ \boldsymbol{c} &= 1 \\ \text{so that}\\ \mathbf{a}\boldsymbol{\beta} &= \boldsymbol{c}\\ \implies \beta_2 + \beta_3 &= 1 \end{align}

## R

This is then a standard quadratic programming problem with a quadratic (in $\boldsymbol{\beta}$) objective function and linear constraints. You can easily solve this using any of the QP packages in R.

Here is an example:

library(quadprog)

# generate some data
mX = cbind(1, matrix(rnorm(100*2), nrow = 100, ncol = 2))
vBeta = c(3, 0.81, 0.19)  # note that the 2nd and 3rd elements add to one
vY = mX %*% vBeta + rnorm(100)

qpStackExchange = solve.QP(Dmat = t(mX)%*% mX,  # X'X
dvec = t(vY) %*% mX,  # Y'X
Amat = matrix(c(0, 1, 1), ncol = 1, nrow = 3),  # matrix a
bvec = 1,  # vector c
meq = 1)  # equality imposed, rather than inequality

qpStackExchange$solution # estimates constrained coefficients qpStackExchange$unconstrained.solution # estimates constrained coefficients


This problem can be formulated as a standard errors-in-variables problem.

Write: $$y - x_2 = a(x_1-x_2) + c$$

now call $z = y-x_2$ and $w = x_1 -x_2$. Then you have the following problem

$$z = aw + c$$ Be careful, Error have changed and are now correlated.

If $\eta_k$ is the error of $x_k$ and $\varepsilon$ is the error for y. Now you have new errors $\phi=\varepsilon -\eta_2$ for $z$ and $\zeta= \eta_1 -\eta_2$ for w.

\begin{align*} \mathrm{Var}[ \phi] & = \mathrm{Var}[\varepsilon]+\mathrm{Var}[\eta_2] -2\mathrm{Cov}[\varepsilon,\eta_2] \\ \mathrm{Var}[ \zeta] & = \mathrm{Var}[\eta_1]+\mathrm{Var}[\eta_2] -2\mathrm{Cov}[\eta_1,\eta_2] \\ \mathrm{Cov}[\phi,\zeta] & = \mathrm{Cov}[\varepsilon -\eta_2,\eta_1 -\eta_2] \\ & = \mathrm{Cov}[\varepsilon,\eta_1] - \mathrm{Cov}[\varepsilon,\eta_2] -\mathrm{Cov}[\eta_2,\eta_1]+ \mathrm{Var}[\eta_2] \end{align*} If all errors are uncorrelated this is: \begin{align*} \mathrm{Var}[ \phi] & = \mathrm{Var}[\varepsilon]+\mathrm{Var}[\eta_2] \\ \mathrm{Var}[ \zeta] & = \mathrm{Var}[\eta_1]+\mathrm{Var}[\eta_2] \\ \mathrm{Cov}[\phi,\zeta] & = \mathrm{Var}[\eta_2] \end{align*} In both cases this problem has a a closed form solution. You can find it Fuller Measurment Error Models (http://www.amazon.com/Measurement-Error-Models-Probability-Statistics/dp/0470095717). I belive it's explained in the introduction chapter.

You can use ConsReg package.

See the example below:

Imagine you want the following constraints in your parameters:

• All coefficients will be less than 1 and greater than -1
• $$x_4 < 0.2$$
• The coefficient of $$x_3$$ and $$x_3^2$$ must satisfied: $$(x_3 + x_3^2 > 0.01$$)

Your can put this constraints to the the function in a easy way:

constraints = '(x3 + I(x3^2)) > .01, x4 < .2'

LOWER = -1, UPPER = 1



And finally, set initial parameters that have to fulfill the constraints above:

ini.pars.coef = c(-.4, .12, -.004, 0.1, 0.1, .15)


Complete example:

require(ConsReg)
data("fake_data")
fit2 = ConsReg(formula = y~x1+x2+x3+ I(x3^2) + x4, data = fake_data,
family = 'gaussian',
constraints = '(x3 + I(x3^2)) > .01, x4 < .2',
optimizer = 'mcmc',
LOWER = -1, UPPER = 1,
ini.pars.coef = c(-.4, .12, -.004, 0.1, 0.1, .15))


• Can you expand on what this code is doing and how it works?
– Sycorax
Oct 18, 2020 at 16:11