Linear regression with constrained coefficient

Question

I have to perform a linear regression of the form $y=ax_1+bx_2+c$ where $b=(1-a)$ and I have errors both on the dependent variable y and on the predictors $x_1, x_2$. I am not sure how to handle the constraint on the second coefficient and the errors on the dependent variable. Any idea?

Answer 1

General constrained OLS problem

Recall that the OLS problem, subject to linear constraints can be written as $$ \begin{align} \arg\min_{\boldsymbol{\beta}}\boldsymbol{Y}'\boldsymbol{Y} - \boldsymbol{Y}'\mathbf{X}\boldsymbol{\beta} - \boldsymbol{\beta}'\mathbf{X}'\boldsymbol{Y} + \boldsymbol{\beta}'\mathbf{X}'\mathbf{X}\boldsymbol{\beta} \end{align}\\ \text{subject to }\quad \mathbf{a}\boldsymbol{\beta} = \boldsymbol{c} $$ where in the general case, $\mathbf{a}$ is a matrix, and $\boldsymbol{c}$ is a vector.

Since the first term does not depend on $\boldsymbol{\beta}$, that we can scale by a constant without changing the solution, and that a scalar is its own transpose, we get $$ \begin{align} \arg\min_{\boldsymbol{\beta}} - \boldsymbol{Y}'\mathbf{X}\boldsymbol{\beta} +\tfrac{1}{2} \boldsymbol{\beta}'\mathbf{X}'\mathbf{X}\boldsymbol{\beta} \end{align}\\ \text{subject to }\quad \mathbf{a}\boldsymbol{\beta} = \boldsymbol{c} $$

Note: I do this so that it maps neatly into the way R solves constrained quadratic programming problems.

Specific case

In your case of three coefficients including the intercept and one constraint, $$ \begin{align} \mathbf{a} &= [0, 1, 1] \\ \boldsymbol{c} &= 1 \\ \text{so that}\\ \mathbf{a}\boldsymbol{\beta} &= \boldsymbol{c}\\ \implies \beta_2 + \beta_3 &= 1 \end{align} $$

R

This is then a standard quadratic programming problem with a quadratic (in $\boldsymbol{\beta}$) objective function and linear constraints. You can easily solve this using any of the QP packages in R.

Here is an example:

library(quadprog)

# generate some data
mX = cbind(1, matrix(rnorm(100*2), nrow = 100, ncol = 2))
vBeta = c(3, 0.81, 0.19)  # note that the 2nd and 3rd elements add to one
vY = mX %*% vBeta + rnorm(100)

# solve the quadratic program
qpStackExchange = solve.QP(Dmat = t(mX)%*% mX,  # X'X
         dvec = t(vY) %*% mX,  # Y'X
         Amat = matrix(c(0, 1, 1), ncol = 1, nrow = 3),  # matrix a
         bvec = 1,  # vector c
         meq = 1)  # equality imposed, rather than inequality

qpStackExchange$solution # estimates constrained coefficients
   qpStackExchange$unconstrained.solution # estimates constrained coefficients

Answer 2

This problem can be formulated as a standard errors-in-variables problem.

Write: \begin{equation} y - x_2 = a(x_1-x_2) + c \end{equation}

now call $z = y-x_2$ and $w = x_1 -x_2$. Then you have the following problem

\begin{equation} z = aw + c \end{equation} Be careful, Error have changed and are now correlated.

If $\eta_k $ is the error of $x_k$ and $\varepsilon$ is the error for y. Now you have new errors $\phi=\varepsilon -\eta_2$ for $z$ and $\zeta= \eta_1 -\eta_2$ for w.

\begin{align*} \mathrm{Var}[ \phi] & = \mathrm{Var}[\varepsilon]+\mathrm{Var}[\eta_2] -2\mathrm{Cov}[\varepsilon,\eta_2] \\ \mathrm{Var}[ \zeta] & = \mathrm{Var}[\eta_1]+\mathrm{Var}[\eta_2] -2\mathrm{Cov}[\eta_1,\eta_2] \\ \mathrm{Cov}[\phi,\zeta] & = \mathrm{Cov}[\varepsilon -\eta_2,\eta_1 -\eta_2] \\ & = \mathrm{Cov}[\varepsilon,\eta_1] - \mathrm{Cov}[\varepsilon,\eta_2] -\mathrm{Cov}[\eta_2,\eta_1]+ \mathrm{Var}[\eta_2] \end{align*} If all errors are uncorrelated this is: \begin{align*} \mathrm{Var}[ \phi] & = \mathrm{Var}[\varepsilon]+\mathrm{Var}[\eta_2] \\ \mathrm{Var}[ \zeta] & = \mathrm{Var}[\eta_1]+\mathrm{Var}[\eta_2] \\ \mathrm{Cov}[\phi,\zeta] & = \mathrm{Var}[\eta_2] \end{align*} In both cases this problem has a a closed form solution. You can find it Fuller Measurment Error Models (http://www.amazon.com/Measurement-Error-Models-Probability-Statistics/dp/0470095717). I belive it's explained in the introduction chapter.

Answer 3

This is an old question, but it may help you.

You can use ConsReg package.

See the example below:

Imagine you want the following constraints in your parameters:

• All coefficients will be less than 1 and greater than -1
• $x_4 < 0.2$
• The coefficient of $x_3$ and $x_3^2$ must satisfied: $(x_3 + x_3^2 > 0.01$)

Your can put this constraints to the the function in a easy way:

constraints = '(x3 + `I(x3^2)`) > .01, x4 < .2'

LOWER = -1, UPPER = 1

And finally, set initial parameters that have to fulfill the constraints above:

ini.pars.coef = c(-.4, .12, -.004, 0.1, 0.1, .15)

Complete example:

require(ConsReg)
data("fake_data")
fit2 = ConsReg(formula = y~x1+x2+x3+ I(x3^2) + x4, data = fake_data,
            family = 'gaussian',
            constraints = '(x3 + `I(x3^2)`) > .01, x4 < .2',
            optimizer = 'mcmc',
            LOWER = -1, UPPER = 1,
            ini.pars.coef = c(-.4, .12, -.004, 0.1, 0.1, .15))