Questions tagged [constrained-regression]
constrained regression is (linear or nonlinear) regression models with further constraints on the coefficients. That could be non-negativity constraints, constraints on the norm of the coefficient vector or otherwise.
81 questions
0
votes
1answer
16 views
polynomial regression in R: how to add hard constraints (go through specified points)? [duplicate]
I'd like to perform polynomial regression on my data (which lies in [0,10]), but I need to ensure that the endpoints of the range are fixed, i.e. that the curve goes through (0,0) and (10,10). So the ...
1
vote
0answers
18 views
nlme ignoring certain control arguments to nlminb
Issue: Do constrained optimization of parameters in nlme::nlme
I'm trying to fit a non-linear mixed effects model using nlme::nlme, which can use 2 optimization schemes: stats::nlminb or stats::nlm.
...
0
votes
1answer
49 views
+50
Confidence limits for constrained penalized log likelihood model
I am estimating parameter $\beta$ as:
\begin{align}
\hat \beta &= \mathop{\mathrm{arg\,max}}_\beta \;\; l(\beta;X,y) - \frac{\lambda}{2}\left(\tilde y-g(\beta,\tilde X)\right)^\prime C^\prime C\...
2
votes
1answer
27 views
Is constrained (nonnegative) least squares a form a regularisation?
Is nonnegative least squares already a form of regularisation? By adding a constraint that $\beta \geq 0$ (the coefficients), does it make sense to add another regularisation term as in LASSO or ridge ...
0
votes
0answers
28 views
Unable to solve using lagrangian multipliers
Suppose
$$K(x,z) = \theta(x)^T \theta(z) = \left\{ \begin{array}{ll}
1 & \text{if } x = z \\
0 & \text{otherwise} \end{array} \right.
$$
and $y_1=+1$ or $-1$.
I ...
5
votes
2answers
120 views
calculating the p-values in a constrained least squares
I have been using Matlab to perform unconstrained least squares (ordinary least squares) and it automatically outputs the coefficients, test statistic and the p-values.
My question is, upon ...
0
votes
0answers
147 views
Constrained Linear Regression with coefficients related by inequality
I have a (slightly simplified) model of the following form:
$Y=c_1X_1 + c_2X_2 + \epsilon$ subject to the constraint $0\leq c_1\leq c_2$.
The distribution of $\epsilon$ is actually not important to ...
0
votes
1answer
60 views
RDA and CCA output- unconstrained inertia 0.00 rank 0
I get an ouput for RDA and for CCA that says that my unconstrained inertia is 0, rank 0. I thought that would be a good thing in the meaning that all the variance in the data is explained by my (...
1
vote
0answers
46 views
Comparing log-transformed and non-log transformed models
Suppose I have the following two models fitted with constrained linear least squares:
Model 1: $Y = \beta_1X_1 + \ldots + \beta_kX_k + \varepsilon$
Model 2: $\log_{10}(Y) = \beta_1\log_{10}(X_1) + \...
3
votes
1answer
24 views
How to perform Least Squares with constraints on a subset of the model coefficients?
For solving an unconstrained LS regression
$$\hat{y}=w_1.x_1+w_2.x_2+w_3.x_3+w_4.x_4 + \epsilon$$
I use the following normal equation:
$$W^*=(X^{\top}X)^{-1}X^{\top}Y $$
If I want to introduce a ...
20
votes
2answers
618 views
The limit of “unit-variance” ridge regression estimator when $\lambda\to\infty$
Consider ridge regression with an additional constraint requiring that $\hat{\mathbf y}$ has unit sum of squares (equivalently, unit variance); if needed, one can assume that $\mathbf y$ has unit sum ...
0
votes
0answers
17 views
Can I find factors, but require linear-combination-of-factors vectors to be all->=0 and sum(factor vector)=1?
Apologies, I'm not a statistician, but this problem came up for a programming project. I really think there are likely huge problems with my nomenclature, but I don't feel confident enough to fix them....
1
vote
0answers
44 views
How to create a constrained spline meta model in R?
Background
I have an outcome measure I'm trying to predict that is on a scale 0-1000. I've trained (say) M different models (model definition doesn't matter) on overlapping sub-ranges of the data (e....
5
votes
1answer
261 views
What are drawbacks of isotonic regression?
I have been reading about isotonic regression and it seems like a great method that will give one a monotone regression function estimator and, moreover, is free of any tuning parameters.
Why are ...
0
votes
0answers
14 views
Hierarchical data, labels sometimes only avaiable per group
I have the following problem:
I have some data, that is hierarchical, i.e. I have single packages, which are grouped in shipments. I have some information about them, such as weight, packaging type, ...
2
votes
0answers
82 views
Fit big GAM with constraints on linear terms
I'm trying to fit MGCV GAM with constraints on linear terms. I feed a dataset with factor variable G. Number of rows is about 30-40k. Number of model parameters - ...
1
vote
0answers
60 views
Choice of ridge parameter in constrained estimation
Apologies for a possible basic question. Suppose I have a model
$$y=X\beta + \epsilon, \quad E[\epsilon|X]=0, \quad X \text{ is } n \times k$$
and, say, matrix $X'X$ is nearly singular and I have a ...
1
vote
1answer
95 views
Fitting a regression with constraints on the relationship between one independent variable and the dependent variable?
Let's say I have a dependent variable y (a rating of the pleasantness of shopping at a particular store from 0 to 100), and 10 independent variables ...
0
votes
1answer
51 views
Estimation with adding-up constraints when the constraint is a coefficient
I have a variable $Y_i = Y_{1,i} + Y_{2,i} + \cdots + Y_{n,i}$. Specifically, $Y_i$ is a measure of consumption for an individual $i$ and $Y_{1,i},Y_{2,i},\cdots$ are consumption expenditures on ...
1
vote
0answers
102 views
R: Constrained coefficients for fitting a linear model to a subset of the data
I fit a regression with 37 variables to my entire dataset and got regression results. One of the variable is the distance in miles (dist). I believe that my data follows two distinct regimes with ...
0
votes
1answer
38 views
Optimization with changing constraints based on parameters to optimize?
I have a large dataset consisting of an ordered categorical variable that is broken into exclusive binary vectors, one for each level of the category.
I would like to find a linear combination of ...
4
votes
1answer
247 views
ridge regression parameter space
In Ridge regression we estimate coefficients as
$\hat{\beta}|\lambda = \arg \min_\beta \|y - X\beta \|_2^2 + \|\lambda \beta\|_2^2 \qquad \qquad(1)$
for a given $\lambda$.
If I wanted the ...
0
votes
0answers
115 views
r - Random slopes regression THROUGH THE ORIGIN (0 intercept)
I am in search of a fixed intercept (through the origin) random slopes model. However, instead of additive errors, I want the SLOPE to be the source of random error.
The problem is this: for a ...
1
vote
1answer
60 views
Constraining a model to output only a range between 1 to 100 (without using logistic regression)
Currently, I am modelling the results of game theory experiments where two parties negotiate to divide a fixed sum of money between themselves. Y is recorded as the percentage of the money which the ...
1
vote
0answers
26 views
Constrain multiple Regression for symmetric Stiffness
I want to do a 3-degree polynomial regression like
$$y_i=a_0+\sum_{u=1}^{k} a_{u}x_{i,u}+\sum_{1\leq u \leq v \leq k} a_{u,v} x_{i,u}x_{i,v}+\sum_{1\leq u \leq v\leq w \leq k}a_{u,v,w}x_{i,u}x_{i,v}...
1
vote
2answers
286 views
Constrained (regularized/damped) solution of system of linear equations in Matlab
I have an overdetermined system of linear equations A*X=B. Where X is a column matrix with N unknowns. The least square solution ...
1
vote
0answers
130 views
NNLS convergence proof
I was trying to understand why the nonnegative least squares (NNLS) algorithm of Lawson & Hanson converges to a solution of
$$
\min f_{0}(x) = \| Cx - d \|^{2}, \\
\text{s.t.} \ x \geq 0.
$$
...
3
votes
3answers
674 views
regression with constraints
I have some domain knowledge I want to use in a regression problem.
Problem statement
The dependent variable $y$ is continuous.
The independent variables are $x_1$ and $x_2$.
Variable $x_1$ is ...
1
vote
1answer
2k views
how to use gradient descent to solve ridge regression with a positivity constraint?
I am working on ridge regression at this moment:
$$\sum{(y_{i}-\beta\cdot X_{i})}^{2} + \lambda \cdot\beta^{2}$$
with an additional constraint:
for some $\beta_{m} > 0 $ and some $\beta_{n}<...
1
vote
1answer
26 views
Restricting model parameters in cumulative logit model (or any ordinal model)
Imagine a model where the linear predictor of the cumulative response probabilities are equal to:
$$y = b_0 + b_1x_1 + b_2x_2 + b_3x_1z_1 + b_4x_2z_1$$
Where $x_1$ and $x_2$ represent a 3 level ...
2
votes
1answer
112 views
Restricted OLS problem
Here is my problem:
Let $1\leq l<k$ be integers and write the design matrix $X$ as $X=[X_1\; X_2]$, where $X_1$ is an $n\times l$-dimensional matrix, and where $X_2$ is an $n\times (k-l)$-...
1
vote
1answer
216 views
Performance of linear least squares regression subject to inequality (bounded interval) constraints on parameters
Consider the following model:
$${\bf y} = {\bf X}{\bf b} + {\bf e}$$
where ${\bf y}, {\bf n}\in {\cal R}^m$, ${\bf b}\in{\cal R}^n$, and ${\bf X}\in{\cal R}^{m \times n}$ where $m>n = {\rm rank}(...
3
votes
1answer
361 views
Linear model with constraints on coefficients in terms of ratios
How to fit a linear model (such as lm() in R) of the form:
$$y = a_1 x_1+ a_2 x_2+ a_3 x_3 + a_4 x_4 + a_5 x_5 + \epsilon$$
with the constraint that : $a_2a_5 = ...
2
votes
0answers
36 views
Reversing constrained PCA Regressions while maintaining constraint [closed]
I have an input matrix X that is 1000x10 and a response Y. To reduce the dimensions, I run PCA, select a certain number of factors (say 2) and transform my X to now be 1000x2. I then scale Y and run ...
1
vote
0answers
109 views
How can I constrain the GLM in R to fit the model between measured upper and lower values?
I have a data set for which the explanatory variable is being compared against two response variables, fairly standard. However, my task is to make the fit of the model be constrained between the ...
4
votes
1answer
2k views
How do I fit a constrained regression in R so all coefficients are positive and above 0 [duplicate]
I am trying to understand how to solve the below quadratic program:
This is my model Y=π1X1+π2X2+π3X3+ε,
My constraints are:
- all weights ...
1
vote
0answers
105 views
Constraining regression coefficients using inequalities the easy way?
I want to do something very simple. I have a regression:
y = b0 + b1*X1 + b2*X2 + b3*X3
and the resulting coefficients must be constrained by:
...
3
votes
1answer
135 views
linear regression with partially known coefficients
I wonder if there some existing work of Linear Regression or Logistic Regression with partially known coefficients ($\beta$).
For a linear regression, $Y=X\beta$, when we already have knowledge ...
2
votes
1answer
425 views
1d linear regression with inequality constraint
I want to solve a simple 1D linear regression problem:
$\mathbf{y} = m \mathbf{x} + c$
such that $m_{low} < m < m_{high}$ and $c_{low} < c < c_{high}$.
How can I solve this problem? ...
0
votes
1answer
181 views
Does constrained EM algorithm work with bad initial inputs?
When trying to perform constrained optimization using EM algorithm, does EM work if the initial solution (x0) violates the constraints?
2
votes
1answer
1k views
Interpreting Canonical Correspondence Analysis (CCA) Inertia - in Vegan
I am wanting to know if I can use the ratio of Constrained/Total Inertia in my CCA to describe 'The variability explained by my constraining variables'. I am asking because I've seen different ...
1
vote
2answers
511 views
How to constrain regression coefficient of two variables to have opposite sign?
I am running a simple linear regression with a few variables but the meanings of the variables are such that certain pairs of variables should have coefficients with opposite signs. How should I ...
5
votes
1answer
1k views
Linear regression polynomial slope constraint in R
My problem is how to find the best decreasing 3rd degree polynomial regression in R.
I have data, lets say
...
3
votes
0answers
685 views
Regularized Linear Regression with specific parameter constraints in R
Using R, I can only find tools for performing L1 and/or L2 regularized linear regression (lars, glmnet) and tools for constrained linear regression (quadprog , or lsei {limSolve} , where the ...
0
votes
1answer
58 views
Estimating $a + bX + cY + bcZ$
How can I estimate functions of the form:
$f(X,Y,Z) = a + bX + cY + bcZ$
I know through expert knowledge that the population coefficient of $Z$ is equal to $bc$ but am not sure how to estimate the ...
1
vote
1answer
90 views
How to choose factors in constrained Linear models?
I was trying to do a linear model analysis where the parameters are constrained (sum to 1 and non-negative).But I found that is not obvious how to apply AIC function(or others) to the parameters ...
3
votes
2answers
113 views
confidence interval in quadratic programming
This is a quadratic programming problem coming from constrained linear regression
$A b + \varepsilon = y$
and
$B^T b >0 $
then minimize $\varepsilon^T \varepsilon=(A b - y)^T (A b - y)$ which is ...
0
votes
0answers
10 views
Using a model with domain/image constraints [duplicate]
I would like to fit a model and the smallest predicted value must be zero. This is what I have done:
...
0
votes
1answer
275 views
Can I set up multiple probability models that sum to 1?
I am interested in consumer research. For simplicity's sake, assume I want to know if people prefer Burger King or McDonald's. My questionnaire will contain a question like "If you had to choose one ...
1
vote
0answers
98 views
Imposing restriction over some regressors in a linear model?
I have the following problem:
Starting with a multivariate linear model $Y=\beta_1x_1+\beta_2x_2+\beta_3x_3+\beta_4x_4+\alpha_1x_5+\alpha_2x_6+\alpha_3x_7+\alpha_4x_8+\epsilon$ and $$X=\begin{pmatrix} ...
