Questions tagged [constrained-regression]

constrained regression is (linear or nonlinear) regression models with further constraints on the coefficients. That could be non-negativity constraints, constraints on the norm of the coefficient vector or otherwise.

Filter by
Sorted by
Tagged with
3
votes
0answers
35 views

Linear regression of higher order polynomial with slope constraint

I am trying to constrain the coefficients on a higher order polynomial (let's say an order 6) for the curve to be decreasing. I have found this link, where the fitting of a 3rd order polynomial is ...
0
votes
0answers
21 views

Inequality Constrained Least Squares: Computation of the Confidence Interval of the mean response

Suppose I wish to find perform inequality constraint least squares on the model $$ y=X\beta +\epsilon$$ subject to a set of constraints $$ T\beta>c \space \space.$$ Is there a way to compute the ...
3
votes
1answer
56 views

How to fit several linear models with linear intercept-slope constrain?

I have a data where several groups exist: $i = 1, 2, 3...n$ ($n$ about 600 groups in total). For group $i$ theory predicts a linear relation $y = a_i + b_i * x$ with $a_i$ and $b_i$ intercept and ...
0
votes
0answers
34 views

Relation between regularization and Lagrange multipliers

This is sort of a follow-up to What is the connection between regularization and the method of lagrange multipliers ? that I thought deserved a new question. Consider an optimization problem of the ...
0
votes
0answers
35 views

Constrained parameters update during hidden Markov model forward-backward algorithm

I want to train a Gaussian hidden Markov model. I currently use the Python package hmmlearn. I looked thouroghly at the code to see how parameters are updated after each training iteration, but I ...
1
vote
1answer
54 views

Adding limits to regression coefficients

For my problem, I have data that contains daily observations of the total time and the volumes of task A completed, task B completed, C, D.. and I am looking to estimate the time it takes to solve ...
0
votes
0answers
34 views

Is there a conjugate prior for normal location family N(x|u,1) such that the mean is always positive?

The conjugate prior to normal location family is usually a normal distribution. However, I want to constrain the mean to be positive. Is there a conjugate prior to the normal location family $x\sim N(...
2
votes
1answer
29 views

Model/link function to deal with dependent variable in range [-1,1]?

My dependent variable, $Y$, contains values anywhere from -1 to 1 (i.e. it is bounded continuously on the range $[-1,1]$). I know that a regular OLS regression on such a variable would sometimes ...
1
vote
0answers
30 views

Sum of residuals in constrained linear regression

Does sum of residuals == 0 hold even if we add ordinality constraints for the coefficients in a regression model? It would be great if someone can point out to any literature (paper/presentation/...
0
votes
0answers
32 views

Closed-form solutions for constrained multiple linear regression

Normally a multiple linear regression is unconstrained $$y=X\beta+\epsilon$$ so that closed-form solutions in the case of data orthogonality ($X^\top X=I$) are $$\beta=(X^\top X)^{-1} X^\top y$$ ...
0
votes
0answers
80 views

Restricted Weighted Linear Regression in R

I have to follwing issue. I would like to run a linear regression imposing a constraint on the weighted coefficients. Let me construct an example: Consider the following cross-sectional regression $...
2
votes
2answers
163 views

Predict a vector of values with constraints? [duplicate]

I am aware of a variety of methods for simultaneously predicting multiple outcomes known sometimes as multivariate regression/analysis. However, my situation is a little more special. I am trying to ...
0
votes
1answer
46 views

How can you constrain values to be positive when fitting a model?

I'm currently fitting a model using maximum likelihood estimation on biological data (electroencephalography). Basically, I'm fitting normal distributions to several subsets of data (experimental ...
0
votes
1answer
39 views

least square with equality constraint and singular matrix

I have the following problem to solve: Ax = b where A is singular. To resolve this I introduce a condition Cx = d Even after this, I am not able to solve it using scipy.optimize.minimize as it ...
0
votes
0answers
19 views

Multilevel modeling, constraint for positive values

I'm currently trying to fit a shifted inverse gaussian to reaction time data (always postive). My paramterization of the model includes 3 parameters, alpha, gamma and tau, which must always be ...
0
votes
1answer
512 views

Is it possible to apply a monotonicity constraint on a Gaussian process regression fit?

Below is a code using scikit-learn where I simply apply Gaussian process regression (GPR) on a set of observed data to produce an expected fit. I know physically that this curve should be ...
1
vote
0answers
235 views

R - Linear Regression with parameters constraint - contr.sum contrast

I am facing a problem with running a simple OLS regression with two categorical independent variable. I would like to impose that the sum of the coefficients referred to one variable (x in the ...
3
votes
0answers
59 views

How to run linear regression with constraints in R? [duplicate]

If I have the following data n<-1000 x1<-rnorm(n,1,1) x2<-rnorm(n,2,2) x3<-rnorm(n,3,3) e<-rnorm(n) y<-3+0.5*x1+0.2*x2+0.3*x3+e I want to fit a ...
1
vote
1answer
142 views

method for predicting a curve

I have data on several curves. the data is of the form: curve_id x y and there are many x/y pairs for each curve and x is limited to some known range. overall, the curves look quite similar in ...
0
votes
1answer
219 views

polynomial regression in R: how to add hard constraints (go through specified points)? [duplicate]

I'd like to perform polynomial regression on my data (which lies in [0,10]), but I need to ensure that the endpoints of the range are fixed, i.e. that the curve goes through (0,0) and (10,10). So the ...
1
vote
0answers
97 views

nlme ignoring certain control arguments to nlminb

Issue: Do constrained optimization of parameters in nlme::nlme I'm trying to fit a non-linear mixed effects model using nlme::nlme, which can use 2 optimization schemes: stats::nlminb or stats::nlm. ...
0
votes
1answer
71 views

Confidence limits for constrained penalized log likelihood model

I am estimating parameter $\beta$ as: \begin{align} \hat \beta &= \mathop{\mathrm{arg\,max}}_\beta \;\; l(\beta;X,y) - \frac{\lambda}{2}\left(\tilde y-g(\beta,\tilde X)\right)^\prime C^\prime C\...
3
votes
1answer
146 views

Is constrained (nonnegative) least squares a form a regularisation?

Is nonnegative least squares already a form of regularisation? By adding a constraint that $\beta \geq 0$ (the coefficients), does it make sense to add another regularisation term as in LASSO or ridge ...
10
votes
3answers
736 views

Calculating the p-values in a constrained (non-negative) least squares

I have been using Matlab to perform unconstrained least squares (ordinary least squares) and it automatically outputs the coefficients, test statistic and the p-values. My question is, upon ...
1
vote
1answer
590 views

Constrained Linear Regression with coefficients related by inequality

I have a (slightly simplified) model of the following form: $Y=c_1X_1 + c_2X_2 + \epsilon$ subject to the constraint $0\leq c_1\leq c_2$. The distribution of $\epsilon$ is actually not important to ...
0
votes
1answer
262 views

RDA and CCA output- unconstrained inertia 0.00 rank 0

I get an ouput for RDA and for CCA that says that my unconstrained inertia is 0, rank 0. I thought that would be a good thing in the meaning that all the variance in the data is explained by my (...
1
vote
0answers
172 views

Comparing log-transformed and non-log transformed models

Suppose I have the following two models fitted with constrained linear least squares: Model 1: $Y = \beta_1X_1 + \ldots + \beta_kX_k + \varepsilon$ Model 2: $\log_{10}(Y) = \beta_1\log_{10}(X_1) + \...
3
votes
1answer
86 views

How to perform Least Squares with constraints on a subset of the model coefficients?

For solving an unconstrained LS regression $$\hat{y}=w_1.x_1+w_2.x_2+w_3.x_3+w_4.x_4 + \epsilon$$ I use the following normal equation: $$W^*=(X^{\top}X)^{-1}X^{\top}Y $$ If I want to introduce a ...
21
votes
2answers
962 views

The limit of “unit-variance” ridge regression estimator when $\lambda\to\infty$

Consider ridge regression with an additional constraint requiring that $\hat{\mathbf y}$ has unit sum of squares (equivalently, unit variance); if needed, one can assume that $\mathbf y$ has unit sum ...
1
vote
0answers
67 views

How to create a constrained spline meta model in R?

Background I have an outcome measure I'm trying to predict that is on a scale 0-1000. I've trained (say) M different models (model definition doesn't matter) on overlapping sub-ranges of the data (e....
5
votes
1answer
839 views

What are drawbacks of isotonic regression?

I have been reading about isotonic regression and it seems like a great method that will give one a monotone regression function estimator and, moreover, is free of any tuning parameters. Why are ...
3
votes
0answers
170 views

Fit big GAM with constraints on linear terms

I'm trying to fit MGCV GAM with constraints on linear terms. I feed a dataset with factor variable G. Number of rows is about 30-40k. Number of model parameters - ...
1
vote
0answers
78 views

Choice of ridge parameter in constrained estimation

Apologies for a possible basic question. Suppose I have a model $$y=X\beta + \epsilon, \quad E[\epsilon|X]=0, \quad X \text{ is } n \times k$$ and, say, matrix $X'X$ is nearly singular and I have a ...
1
vote
1answer
131 views

Fitting a regression with constraints on the relationship between one independent variable and the dependent variable?

Let's say I have a dependent variable y (a rating of the pleasantness of shopping at a particular store from 0 to 100), and 10 independent variables ...
0
votes
1answer
99 views

Estimation with adding-up constraints when the constraint is a coefficient

I have a variable $Y_i = Y_{1,i} + Y_{2,i} + \cdots + Y_{n,i}$. Specifically, $Y_i$ is a measure of consumption for an individual $i$ and $Y_{1,i},Y_{2,i},\cdots$ are consumption expenditures on ...
1
vote
0answers
125 views

R: Constrained coefficients for fitting a linear model to a subset of the data

I fit a regression with 37 variables to my entire dataset and got regression results. One of the variable is the distance in miles (dist). I believe that my data follows two distinct regimes with ...
0
votes
1answer
52 views

Optimization with changing constraints based on parameters to optimize?

I have a large dataset consisting of an ordered categorical variable that is broken into exclusive binary vectors, one for each level of the category. I would like to find a linear combination of ...
4
votes
1answer
347 views

ridge regression parameter space

In Ridge regression we estimate coefficients as $\hat{\beta}|\lambda = \arg \min_\beta \|y - X\beta \|_2^2 + \|\lambda \beta\|_2^2 \qquad \qquad(1)$ for a given $\lambda$. If I wanted the ...
0
votes
0answers
217 views

r - Random slopes regression THROUGH THE ORIGIN (0 intercept)

I am in search of a fixed intercept (through the origin) random slopes model. However, instead of additive errors, I want the SLOPE to be the source of random error. The problem is this: for a ...
1
vote
1answer
87 views

Constraining a model to output only a range between 1 to 100 (without using logistic regression)

Currently, I am modelling the results of game theory experiments where two parties negotiate to divide a fixed sum of money between themselves. Y is recorded as the percentage of the money which the ...
1
vote
0answers
28 views

Constrain multiple Regression for symmetric Stiffness

I want to do a 3-degree polynomial regression like $$y_i=a_0+\sum_{u=1}^{k} a_{u}x_{i,u}+\sum_{1\leq u \leq v \leq k} a_{u,v} x_{i,u}x_{i,v}+\sum_{1\leq u \leq v\leq w \leq k}a_{u,v,w}x_{i,u}x_{i,v}...
2
votes
2answers
487 views

Constrained (regularized/damped) solution of system of linear equations in Matlab

I have an overdetermined system of linear equations A*X=B. Where X is a column matrix with N unknowns. The least square solution ...
2
votes
0answers
186 views

NNLS convergence proof

I was trying to understand why the nonnegative least squares (NNLS) algorithm of Lawson & Hanson converges to a solution of $$ \min f_{0}(x) = \| Cx - d \|^{2}, \\ \text{s.t.} \ x \geq 0. $$ ...
5
votes
3answers
1k views

regression with constraints

I have some domain knowledge I want to use in a regression problem. Problem statement The dependent variable $y$ is continuous. The independent variables are $x_1$ and $x_2$. Variable $x_1$ is ...
1
vote
1answer
3k views

how to use gradient descent to solve ridge regression with a positivity constraint?

I am working on ridge regression at this moment: $$\sum{(y_{i}-\beta\cdot X_{i})}^{2} + \lambda \cdot\beta^{2}$$ with an additional constraint: for some $\beta_{m} > 0 $ and some $\beta_{n}<...
1
vote
1answer
30 views

Restricting model parameters in cumulative logit model (or any ordinal model)

Imagine a model where the linear predictor of the cumulative response probabilities are equal to: $$y = b_0 + b_1x_1 + b_2x_2 + b_3x_1z_1 + b_4x_2z_1$$ Where $x_1$ and $x_2$ represent a 3 level ...
2
votes
1answer
191 views

Restricted OLS problem

Here is my problem: Let $1\leq l<k$ be integers and write the design matrix $X$ as $X=[X_1\; X_2]$, where $X_1$ is an $n\times l$-dimensional matrix, and where $X_2$ is an $n\times (k-l)$-...
2
votes
1answer
286 views

Performance of linear least squares regression subject to inequality (bounded interval) constraints on parameters

Consider the following model: $${\bf y} = {\bf X}{\bf b} + {\bf e}$$ where ${\bf y}, {\bf n}\in {\cal R}^m$, ${\bf b}\in{\cal R}^n$, and ${\bf X}\in{\cal R}^{m \times n}$ where $m>n = {\rm rank}(...
4
votes
1answer
451 views

Linear model with constraints on coefficients in terms of ratios

How to fit a linear model (such as lm() in R) of the form: $$y = a_1 x_1+ a_2 x_2+ a_3 x_3 + a_4 x_4 + a_5 x_5 + \epsilon$$ with the constraint that : $a_2a_5 = ...
2
votes
0answers
39 views

Reversing constrained PCA Regressions while maintaining constraint [closed]

I have an input matrix X that is 1000x10 and a response Y. To reduce the dimensions, I run PCA, select a certain number of factors (say 2) and transform my X to now be 1000x2. I then scale Y and run ...