Questions tagged [constrained-regression]
constrained regression is (linear or nonlinear) regression models with further constraints on the coefficients. That could be non-negativity constraints, constraints on the norm of the coefficient vector or otherwise.
96
questions
3
votes
0answers
35 views
Linear regression of higher order polynomial with slope constraint
I am trying to constrain the coefficients on a higher order polynomial (let's say an order 6) for the curve to be decreasing.
I have found this link, where the fitting of a 3rd order polynomial is ...
0
votes
0answers
21 views
Inequality Constrained Least Squares: Computation of the Confidence Interval of the mean response
Suppose I wish to find perform inequality constraint least squares on the model
$$ y=X\beta +\epsilon$$
subject to a set of constraints
$$ T\beta>c \space \space.$$
Is there a way to compute the ...
3
votes
1answer
56 views
How to fit several linear models with linear intercept-slope constrain?
I have a data where several groups exist: $i = 1, 2, 3...n$ ($n$ about 600 groups in total).
For group $i$ theory predicts a linear relation $y = a_i + b_i * x$ with $a_i$ and $b_i$ intercept and ...
0
votes
0answers
34 views
Relation between regularization and Lagrange multipliers
This is sort of a follow-up to What is the connection between regularization and the method of lagrange multipliers ? that I thought deserved a new question.
Consider an optimization problem of the ...
0
votes
0answers
35 views
Constrained parameters update during hidden Markov model forward-backward algorithm
I want to train a Gaussian hidden Markov model.
I currently use the Python package hmmlearn.
I looked thouroghly at the code to see how parameters are updated after each training iteration, but I ...
1
vote
1answer
54 views
Adding limits to regression coefficients
For my problem, I have data that contains daily observations of the total time and the volumes of task A completed, task B completed, C, D.. and I am looking to estimate the time it takes to solve ...
0
votes
0answers
34 views
Is there a conjugate prior for normal location family N(x|u,1) such that the mean is always positive?
The conjugate prior to normal location family is usually a normal distribution. However, I want to constrain the mean to be positive.
Is there a conjugate prior to the normal location family $x\sim N(...
2
votes
1answer
29 views
Model/link function to deal with dependent variable in range [-1,1]?
My dependent variable, $Y$, contains values anywhere from -1 to 1 (i.e. it is bounded continuously on the range $[-1,1]$). I know that a regular OLS regression on such a variable would sometimes ...
1
vote
0answers
30 views
Sum of residuals in constrained linear regression
Does sum of residuals == 0 hold even if we add ordinality constraints for the coefficients in a regression model?
It would be great if someone can point out to any literature (paper/presentation/...
0
votes
0answers
32 views
Closed-form solutions for constrained multiple linear regression
Normally a multiple linear regression is unconstrained
$$y=X\beta+\epsilon$$
so that closed-form solutions in the case of data orthogonality ($X^\top X=I$) are
$$\beta=(X^\top X)^{-1} X^\top y$$
...
0
votes
0answers
80 views
Restricted Weighted Linear Regression in R
I have to follwing issue. I would like to run a linear regression imposing a constraint on the weighted coefficients. Let me construct an example:
Consider the following cross-sectional regression
$...
2
votes
2answers
163 views
Predict a vector of values with constraints? [duplicate]
I am aware of a variety of methods for simultaneously predicting multiple outcomes known sometimes as multivariate regression/analysis. However, my situation is a little more special. I am trying to ...
0
votes
1answer
46 views
How can you constrain values to be positive when fitting a model?
I'm currently fitting a model using maximum likelihood estimation on biological data (electroencephalography). Basically, I'm fitting normal distributions to several subsets of data (experimental ...
0
votes
1answer
39 views
least square with equality constraint and singular matrix
I have the following problem to solve:
Ax = b where A is singular.
To resolve this I introduce a condition Cx = d
Even after this, I am not able to solve it using scipy.optimize.minimize as it ...
0
votes
0answers
19 views
Multilevel modeling, constraint for positive values
I'm currently trying to fit a shifted inverse gaussian to reaction time data (always postive). My paramterization of the model includes 3 parameters, alpha, gamma and tau, which must always be ...
0
votes
1answer
512 views
Is it possible to apply a monotonicity constraint on a Gaussian process regression fit?
Below is a code using scikit-learn where I simply apply Gaussian process regression (GPR) on a set of observed data to produce an expected fit. I know physically that this curve should be ...
1
vote
0answers
235 views
R - Linear Regression with parameters constraint - contr.sum contrast
I am facing a problem with running a simple OLS regression with two categorical independent variable.
I would like to impose that the sum of the coefficients referred to one variable (x in the ...
3
votes
0answers
59 views
How to run linear regression with constraints in R? [duplicate]
If I have the following data
n<-1000
x1<-rnorm(n,1,1)
x2<-rnorm(n,2,2)
x3<-rnorm(n,3,3)
e<-rnorm(n)
y<-3+0.5*x1+0.2*x2+0.3*x3+e
I want to fit a ...
1
vote
1answer
142 views
method for predicting a curve
I have data on several curves. the data is of the form:
curve_id x y
and there are many x/y pairs for each curve and x is limited to some known range. overall, the curves look quite similar in ...
0
votes
1answer
219 views
polynomial regression in R: how to add hard constraints (go through specified points)? [duplicate]
I'd like to perform polynomial regression on my data (which lies in [0,10]), but I need to ensure that the endpoints of the range are fixed, i.e. that the curve goes through (0,0) and (10,10). So the ...
1
vote
0answers
97 views
nlme ignoring certain control arguments to nlminb
Issue: Do constrained optimization of parameters in nlme::nlme
I'm trying to fit a non-linear mixed effects model using nlme::nlme, which can use 2 optimization schemes: stats::nlminb or stats::nlm.
...
0
votes
1answer
71 views
Confidence limits for constrained penalized log likelihood model
I am estimating parameter $\beta$ as:
\begin{align}
\hat \beta &= \mathop{\mathrm{arg\,max}}_\beta \;\; l(\beta;X,y) - \frac{\lambda}{2}\left(\tilde y-g(\beta,\tilde X)\right)^\prime C^\prime C\...
3
votes
1answer
146 views
Is constrained (nonnegative) least squares a form a regularisation?
Is nonnegative least squares already a form of regularisation? By adding a constraint that $\beta \geq 0$ (the coefficients), does it make sense to add another regularisation term as in LASSO or ridge ...
10
votes
3answers
736 views
Calculating the p-values in a constrained (non-negative) least squares
I have been using Matlab to perform unconstrained least squares (ordinary least squares) and it automatically outputs the coefficients, test statistic and the p-values.
My question is, upon ...
1
vote
1answer
590 views
Constrained Linear Regression with coefficients related by inequality
I have a (slightly simplified) model of the following form:
$Y=c_1X_1 + c_2X_2 + \epsilon$ subject to the constraint $0\leq c_1\leq c_2$.
The distribution of $\epsilon$ is actually not important to ...
0
votes
1answer
262 views
RDA and CCA output- unconstrained inertia 0.00 rank 0
I get an ouput for RDA and for CCA that says that my unconstrained inertia is 0, rank 0. I thought that would be a good thing in the meaning that all the variance in the data is explained by my (...
1
vote
0answers
172 views
Comparing log-transformed and non-log transformed models
Suppose I have the following two models fitted with constrained linear least squares:
Model 1: $Y = \beta_1X_1 + \ldots + \beta_kX_k + \varepsilon$
Model 2: $\log_{10}(Y) = \beta_1\log_{10}(X_1) + \...
3
votes
1answer
86 views
How to perform Least Squares with constraints on a subset of the model coefficients?
For solving an unconstrained LS regression
$$\hat{y}=w_1.x_1+w_2.x_2+w_3.x_3+w_4.x_4 + \epsilon$$
I use the following normal equation:
$$W^*=(X^{\top}X)^{-1}X^{\top}Y $$
If I want to introduce a ...
21
votes
2answers
962 views
The limit of “unit-variance” ridge regression estimator when $\lambda\to\infty$
Consider ridge regression with an additional constraint requiring that $\hat{\mathbf y}$ has unit sum of squares (equivalently, unit variance); if needed, one can assume that $\mathbf y$ has unit sum ...
asked Apr 12 '18 at 12:22
amoeba says Reinstate Monica
76.5k20 gold badges238 silver badges286 bronze badges
1
vote
0answers
67 views
How to create a constrained spline meta model in R?
Background
I have an outcome measure I'm trying to predict that is on a scale 0-1000. I've trained (say) M different models (model definition doesn't matter) on overlapping sub-ranges of the data (e....
5
votes
1answer
839 views
What are drawbacks of isotonic regression?
I have been reading about isotonic regression and it seems like a great method that will give one a monotone regression function estimator and, moreover, is free of any tuning parameters.
Why are ...
3
votes
0answers
170 views
Fit big GAM with constraints on linear terms
I'm trying to fit MGCV GAM with constraints on linear terms. I feed a dataset with factor variable G. Number of rows is about 30-40k. Number of model parameters - ...
1
vote
0answers
78 views
Choice of ridge parameter in constrained estimation
Apologies for a possible basic question. Suppose I have a model
$$y=X\beta + \epsilon, \quad E[\epsilon|X]=0, \quad X \text{ is } n \times k$$
and, say, matrix $X'X$ is nearly singular and I have a ...
1
vote
1answer
131 views
Fitting a regression with constraints on the relationship between one independent variable and the dependent variable?
Let's say I have a dependent variable y (a rating of the pleasantness of shopping at a particular store from 0 to 100), and 10 independent variables ...
0
votes
1answer
99 views
Estimation with adding-up constraints when the constraint is a coefficient
I have a variable $Y_i = Y_{1,i} + Y_{2,i} + \cdots + Y_{n,i}$. Specifically, $Y_i$ is a measure of consumption for an individual $i$ and $Y_{1,i},Y_{2,i},\cdots$ are consumption expenditures on ...
1
vote
0answers
125 views
R: Constrained coefficients for fitting a linear model to a subset of the data
I fit a regression with 37 variables to my entire dataset and got regression results. One of the variable is the distance in miles (dist). I believe that my data follows two distinct regimes with ...
0
votes
1answer
52 views
Optimization with changing constraints based on parameters to optimize?
I have a large dataset consisting of an ordered categorical variable that is broken into exclusive binary vectors, one for each level of the category.
I would like to find a linear combination of ...
4
votes
1answer
347 views
ridge regression parameter space
In Ridge regression we estimate coefficients as
$\hat{\beta}|\lambda = \arg \min_\beta \|y - X\beta \|_2^2 + \|\lambda \beta\|_2^2 \qquad \qquad(1)$
for a given $\lambda$.
If I wanted the ...
0
votes
0answers
217 views
r - Random slopes regression THROUGH THE ORIGIN (0 intercept)
I am in search of a fixed intercept (through the origin) random slopes model. However, instead of additive errors, I want the SLOPE to be the source of random error.
The problem is this: for a ...
1
vote
1answer
87 views
Constraining a model to output only a range between 1 to 100 (without using logistic regression)
Currently, I am modelling the results of game theory experiments where two parties negotiate to divide a fixed sum of money between themselves. Y is recorded as the percentage of the money which the ...
1
vote
0answers
28 views
Constrain multiple Regression for symmetric Stiffness
I want to do a 3-degree polynomial regression like
$$y_i=a_0+\sum_{u=1}^{k} a_{u}x_{i,u}+\sum_{1\leq u \leq v \leq k} a_{u,v} x_{i,u}x_{i,v}+\sum_{1\leq u \leq v\leq w \leq k}a_{u,v,w}x_{i,u}x_{i,v}...
2
votes
2answers
487 views
Constrained (regularized/damped) solution of system of linear equations in Matlab
I have an overdetermined system of linear equations A*X=B. Where X is a column matrix with N unknowns. The least square solution ...
2
votes
0answers
186 views
NNLS convergence proof
I was trying to understand why the nonnegative least squares (NNLS) algorithm of Lawson & Hanson converges to a solution of
$$
\min f_{0}(x) = \| Cx - d \|^{2}, \\
\text{s.t.} \ x \geq 0.
$$
...
5
votes
3answers
1k views
regression with constraints
I have some domain knowledge I want to use in a regression problem.
Problem statement
The dependent variable $y$ is continuous.
The independent variables are $x_1$ and $x_2$.
Variable $x_1$ is ...
1
vote
1answer
3k views
how to use gradient descent to solve ridge regression with a positivity constraint?
I am working on ridge regression at this moment:
$$\sum{(y_{i}-\beta\cdot X_{i})}^{2} + \lambda \cdot\beta^{2}$$
with an additional constraint:
for some $\beta_{m} > 0 $ and some $\beta_{n}<...
1
vote
1answer
30 views
Restricting model parameters in cumulative logit model (or any ordinal model)
Imagine a model where the linear predictor of the cumulative response probabilities are equal to:
$$y = b_0 + b_1x_1 + b_2x_2 + b_3x_1z_1 + b_4x_2z_1$$
Where $x_1$ and $x_2$ represent a 3 level ...
2
votes
1answer
191 views
Restricted OLS problem
Here is my problem:
Let $1\leq l<k$ be integers and write the design matrix $X$ as $X=[X_1\; X_2]$, where $X_1$ is an $n\times l$-dimensional matrix, and where $X_2$ is an $n\times (k-l)$-...
2
votes
1answer
286 views
Performance of linear least squares regression subject to inequality (bounded interval) constraints on parameters
Consider the following model:
$${\bf y} = {\bf X}{\bf b} + {\bf e}$$
where ${\bf y}, {\bf n}\in {\cal R}^m$, ${\bf b}\in{\cal R}^n$, and ${\bf X}\in{\cal R}^{m \times n}$ where $m>n = {\rm rank}(...
4
votes
1answer
451 views
Linear model with constraints on coefficients in terms of ratios
How to fit a linear model (such as lm() in R) of the form:
$$y = a_1 x_1+ a_2 x_2+ a_3 x_3 + a_4 x_4 + a_5 x_5 + \epsilon$$
with the constraint that : $a_2a_5 = ...
2
votes
0answers
39 views
Reversing constrained PCA Regressions while maintaining constraint [closed]
I have an input matrix X that is 1000x10 and a response Y. To reduce the dimensions, I run PCA, select a certain number of factors (say 2) and transform my X to now be 1000x2. I then scale Y and run ...
