7
$\begingroup$

I have to perform a linear regression of the form $y=ax_1+bx_2+c$ where $b=(1-a)$ and I have errors both on the dependent variable y and on the predictors $x_1, x_2$. I am not sure how to handle the constraint on the second coefficient and the errors on the dependent variable. Any idea?

$\endgroup$
3
  • 3
    $\begingroup$ Your constraint implies that you are regressing $y$ on a single variable $x_1 + x_2$ and forcing its coefficient to be $1$. That doesn't solve the problem of errors in predictors. Errors in the dependent variable are what you expect with regression. $\endgroup$
    – Nick Cox
    Dec 9, 2013 at 13:32
  • $\begingroup$ Ok, thanks. But I am not sure how should I force the coefficient of (x1+x2) to be 1.. Can I do that with a least square fitting? $\endgroup$
    – giuliac
    Dec 9, 2013 at 13:49
  • 2
    $\begingroup$ You can constrain things with least squares by reparameterizing (let y* = y-x2 and x* = x1-x2. Fit y* = ax* + c and then let b = 1-a. However, that won't deal with your errors-in-variables. In some cases at least, though, reparameterizing might work there as well. $\endgroup$
    – Glen_b
    Dec 9, 2013 at 14:18

3 Answers 3

9
$\begingroup$

General constrained OLS problem

Recall that the OLS problem, subject to linear constraints can be written as $$ \begin{align} \arg\min_{\boldsymbol{\beta}}\boldsymbol{Y}'\boldsymbol{Y} - \boldsymbol{Y}'\mathbf{X}\boldsymbol{\beta} - \boldsymbol{\beta}'\mathbf{X}'\boldsymbol{Y} + \boldsymbol{\beta}'\mathbf{X}'\mathbf{X}\boldsymbol{\beta} \end{align}\\ \text{subject to }\quad \mathbf{a}\boldsymbol{\beta} = \boldsymbol{c} $$ where in the general case, $\mathbf{a}$ is a matrix, and $\boldsymbol{c}$ is a vector.

Since the first term does not depend on $\boldsymbol{\beta}$, that we can scale by a constant without changing the solution, and that a scalar is its own transpose, we get $$ \begin{align} \arg\min_{\boldsymbol{\beta}} - \boldsymbol{Y}'\mathbf{X}\boldsymbol{\beta} +\tfrac{1}{2} \boldsymbol{\beta}'\mathbf{X}'\mathbf{X}\boldsymbol{\beta} \end{align}\\ \text{subject to }\quad \mathbf{a}\boldsymbol{\beta} = \boldsymbol{c} $$

Note: I do this so that it maps neatly into the way R solves constrained quadratic programming problems.

Specific case

In your case of three coefficients including the intercept and one constraint, $$ \begin{align} \mathbf{a} &= [0, 1, 1] \\ \boldsymbol{c} &= 1 \\ \text{so that}\\ \mathbf{a}\boldsymbol{\beta} &= \boldsymbol{c}\\ \implies \beta_2 + \beta_3 &= 1 \end{align} $$

R

This is then a standard quadratic programming problem with a quadratic (in $\boldsymbol{\beta}$) objective function and linear constraints. You can easily solve this using any of the QP packages in R.

Here is an example:

library(quadprog)

# generate some data
mX = cbind(1, matrix(rnorm(100*2), nrow = 100, ncol = 2))
vBeta = c(3, 0.81, 0.19)  # note that the 2nd and 3rd elements add to one
vY = mX %*% vBeta + rnorm(100)

# solve the quadratic program
qpStackExchange = solve.QP(Dmat = t(mX)%*% mX,  # X'X
         dvec = t(vY) %*% mX,  # Y'X
         Amat = matrix(c(0, 1, 1), ncol = 1, nrow = 3),  # matrix a
         bvec = 1,  # vector c
         meq = 1)  # equality imposed, rather than inequality

qpStackExchange$solution # estimates constrained coefficients
   qpStackExchange$unconstrained.solution # estimates constrained coefficients
$\endgroup$
3
$\begingroup$

This problem can be formulated as a standard errors-in-variables problem.

Write: \begin{equation} y - x_2 = a(x_1-x_2) + c \end{equation}

now call $z = y-x_2$ and $w = x_1 -x_2$. Then you have the following problem

\begin{equation} z = aw + c \end{equation} Be careful, Error have changed and are now correlated.

If $\eta_k $ is the error of $x_k$ and $\varepsilon$ is the error for y. Now you have new errors $\phi=\varepsilon -\eta_2$ for $z$ and $\zeta= \eta_1 -\eta_2$ for w.

\begin{align*} \mathrm{Var}[ \phi] & = \mathrm{Var}[\varepsilon]+\mathrm{Var}[\eta_2] -2\mathrm{Cov}[\varepsilon,\eta_2] \\ \mathrm{Var}[ \zeta] & = \mathrm{Var}[\eta_1]+\mathrm{Var}[\eta_2] -2\mathrm{Cov}[\eta_1,\eta_2] \\ \mathrm{Cov}[\phi,\zeta] & = \mathrm{Cov}[\varepsilon -\eta_2,\eta_1 -\eta_2] \\ & = \mathrm{Cov}[\varepsilon,\eta_1] - \mathrm{Cov}[\varepsilon,\eta_2] -\mathrm{Cov}[\eta_2,\eta_1]+ \mathrm{Var}[\eta_2] \end{align*} If all errors are uncorrelated this is: \begin{align*} \mathrm{Var}[ \phi] & = \mathrm{Var}[\varepsilon]+\mathrm{Var}[\eta_2] \\ \mathrm{Var}[ \zeta] & = \mathrm{Var}[\eta_1]+\mathrm{Var}[\eta_2] \\ \mathrm{Cov}[\phi,\zeta] & = \mathrm{Var}[\eta_2] \end{align*} In both cases this problem has a a closed form solution. You can find it Fuller Measurment Error Models (http://www.amazon.com/Measurement-Error-Models-Probability-Statistics/dp/0470095717). I belive it's explained in the introduction chapter.

$\endgroup$
2
$\begingroup$

This is an old question, but it may help you.

You can use ConsReg package.

See the example below:

Imagine you want the following constraints in your parameters:

  • All coefficients will be less than 1 and greater than -1
  • $x_4 < 0.2$
  • The coefficient of $x_3$ and $x_3^2$ must satisfied: $(x_3 + x_3^2 > 0.01$)

Your can put this constraints to the the function in a easy way:

constraints = '(x3 + `I(x3^2)`) > .01, x4 < .2'

LOWER = -1, UPPER = 1

And finally, set initial parameters that have to fulfill the constraints above:

ini.pars.coef = c(-.4, .12, -.004, 0.1, 0.1, .15)

Complete example:

require(ConsReg)
data("fake_data")
fit2 = ConsReg(formula = y~x1+x2+x3+ I(x3^2) + x4, data = fake_data,
            family = 'gaussian',
            constraints = '(x3 + `I(x3^2)`) > .01, x4 < .2',
            optimizer = 'mcmc',
            LOWER = -1, UPPER = 1,
            ini.pars.coef = c(-.4, .12, -.004, 0.1, 0.1, .15))

$\endgroup$
1
  • $\begingroup$ Can you expand on what this code is doing and how it works? $\endgroup$
    – Sycorax
    Oct 18, 2020 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.