Questions tagged [constrained-regression]

constrained regression is (linear or nonlinear) regression models with further constraints on the coefficients. That could be non-negativity constraints, constraints on the norm of the coefficient vector or otherwise.

Filter by
Sorted by
Tagged with
0 votes
1 answer
26 views

Shrinkage methods: does non-convex constraints ($q < 1$) also induce sparsity?

I'm reading ESLII, in particular the chapter about shrinkage methods, lasso, and ridge regression. The optimal model parameters for a given constraint $\sum | \beta_j |^q < \alpha $ are given by ...
user avatar
  • 155
0 votes
0 answers
18 views

Confidence intervals for non-negative least squares

Can we use the non-parametric bootstrapping to compute the confidence intervals for the regression coefficients estimated from non-negative least squares? I wonder whether this has the same issues ...
user avatar
  • 601
0 votes
0 answers
13 views

Recover Primal Linear Programming Solution from Dual with LAD Regression?

This link discusses different ways of writing a classic LAD regression with a linear program. The classic way of writing LAD regression ($y = X \beta + r$) as a linear program is \begin{equation} \...
user avatar
3 votes
1 answer
23 views

Allow gradient descent to go beyond constraint but punish for it

Disclaimer: I considered posting this on mathSE but thought maybe this is more fitting here (no pun intended). Status Quo I have a list of data points $\{x_k, y_k\}$ and a set of functions $f_i(x)$ ...
user avatar
1 vote
0 answers
27 views

Add constraints to linear regression which make some of the coefficients positive [duplicate]

I run a linear model which is about sales~ Investments, denote the Investments as I and ...
user avatar
0 votes
1 answer
18 views

Testing for plateau with discrete timepoints

I have measured a continuous outcome at 4 timepoints (t1, t2, t3, t4), spaced evenly apart. The measurements at each time point are on different groups (i.e. the responses at t1 come from all ...
user avatar
  • 1
4 votes
1 answer
79 views

Linear model with constraints in R

I have the following problem: I have a simple linear model with 3 independent variables. ...
user avatar
1 vote
1 answer
123 views

Fit points with a continuous, piecewise linear function with minimum number of points for each segment

I have a set of two-dimensional data points that I want to fit with a continuous piecewise linear function with one break point. However, I want each of the two segments to be supported by a minimum ...
user avatar
  • 13
2 votes
0 answers
28 views

Covariance matrix of beta coefficients for constrained multiple regression

I have a linear least-squares problem with constraints that two of the coefficients must be non-negative. For a typical (unconstrained) least squares estimation, I know that the variance-covariance ...
user avatar
  • 21
0 votes
0 answers
43 views

Why SCM weights *should* sum to 1?

When we use synthetic controls, we consider $j=2, \dots, J+1$ units across $t \in \{T_{-} \ldots T_0 \ldots T_{+}\} \in \mathbb{Z}$ pre/post event-time periods. Abadie et. al. make the point that to ...
user avatar
0 votes
0 answers
36 views

How to Model Constrained Outcome in Multivariate Regression Context

I have a problem in which the ratio of two different outcomes as a function of time $\frac{Y_{1t}}{Y_{2t}}$ is unknown constant $c$. I would like to estimate regression models of the following type: $$...
user avatar
  • 1
1 vote
1 answer
75 views

How do linear constraints affect the convexity of my OLS-like optimisation problem?

I would like to augment a linear regression (so a convex OLS problem) with some additional constraints on the coefficients to match the subject I'm working on. Having $x\in \mathbb{R}^n$, the solution ...
user avatar
  • 11
0 votes
1 answer
52 views

Minimizing $L_2$ norm with constrained residual sum of squares (RSS)

I have some complex-valued time-series data, $y \in \mathbb{C}^n$ - a signal with additive Gaussian white noise. The goal is to find the Fourier coefficients of this signal. Ideally, you would just do ...
user avatar
  • 163
0 votes
0 answers
75 views

How to get standard errors of linear regression coefficients subject to linear equality constraints

I have a regression problem with multiple sets of sum-to-one categorical features (e.g. X1=Male, X2=Female, X3=Weekday, X4=Weekend), intercept needs to be included as well as some restrictions on the ...
user avatar
0 votes
0 answers
48 views

Neural network regression with a constrained dependent variable

I'm using a multi-layer perceptron model to predict the cumulative number of boardings and alightings from a bus travelling on a standard route based on previous cumulative boarding and alighting ...
user avatar
1 vote
0 answers
52 views

Difference between glmnet and nnls for non-negative least squares in R

I'm trying to do some non-negative linear regressions in R, and I found in the blog here https://www.r-bloggers.com/2019/11/non-negative-least-squares/ that either the package ...
user avatar
  • 11
3 votes
2 answers
423 views

How to fit a constrained regression in R?

My regression: $Y = α + β_1X_1 + β_2X_2 + β_3X_3 + β_4X_4 + β_5X_5 + β_6X_6 + β_7X_7 + β_8X_8 + β_9X_9 + β_{10}X_{10} + β_{11}X_{11} + β_{12}X_{12} + β_{13}X_{13} + β_{14}X_{14} + ε$ Constraints that ...
user avatar
  • 31
0 votes
1 answer
231 views

Can it be proven that non-negative least squares is a perfectly convex problem?

Can it be proven that NNLS is a perfectly convex problem? Myre writes in section 3 of his TNT-NN manuscript: The NNLS objective function is a convex quadratic function with linear inequalities as ...
user avatar
  • 181
1 vote
1 answer
50 views

Numerical solution to the constrained ridge regression

The constrained ridge regression problem is of the form: $\arg\min_{\|\beta\|_2\le t}\|X\beta-y\|_2$. Given a matrix $X$, a vector $y$ and the constrain parameter $t$, how do you solve it numerically? ...
user avatar
  • 11
0 votes
0 answers
29 views

Constraining regression coefficient to non-negative [duplicate]

I have a regression problem where I don't want the coefficients to be negative. Is setting negative coefficients of OLS to zero the same as constraining the coefficient to be non-zero and solving it ...
user avatar
  • 761
2 votes
0 answers
45 views

How to get accurate estimates on Neural Networks Hessian?

I need to get not only accurate estimates on the neural network output itself but also on its second order derivatives in order to use the NN for optimization problems. With Adam optimizer I can't get ...
user avatar
1 vote
0 answers
167 views

Linear Regression with constraint(s): need s.e., t-stat, and p-values of the regressors

I am performing a linear regression and what I need is (1) to constrain the sum of the regression coefficients to 1, and (2) to constrain the sum of regression coefficients to 1 AND each regression ...
user avatar
0 votes
0 answers
42 views

Perform regression on a set of equations with additional constraints

I want to analyse the composition of a ternary mixture (a mixture containing of three components) by combining measurements of the mixture's density (denoted $\rho$) and its refractive index (denoted $...
user avatar
  • 101
0 votes
0 answers
19 views

Can regression coefficients be guaranteed to be normalized?

When solving a multivariate linear regression problem of the form $A\vec{x}=\vec{y}$ where $A$ and $\vec{y}$ are known. Is there any from of preprocessing or scaling that can be done $A$ and or $\vec{...
user avatar
3 votes
3 answers
2k views

How to test if sum of two coefficients of ols model is greater than zero using R?

The regression model is: y = b0 +b1x1 + b2x2 + b3x3 + e I want to test if b1 + b2 > 0. the R package ...
user avatar
  • 33
2 votes
1 answer
54 views

How can this model be fit using R's lm function?

I have data that consists of a grouping variable $\tt{grp}$, a predictor $\tt{x}$, and a response $\tt{y}$. There are three groups. I want to fit to this data the model $y = \alpha_i + (\beta + \...
user avatar
  • 251
0 votes
1 answer
60 views

Constrained optimization of joint Bernoulli density parametrized by a neural network

I want to learn a joint distribution on $n$ Bernoulli random variables conditioned on a random variable $b\sim D$ and parametrized by a neural network, $f$: $$p(A = (a_1, \ldots, a_n)|b) = f_{\Lambda}(...
user avatar
  • 151
3 votes
0 answers
36 views

Regression with coefficients having a multiplicative relationship

Assume I need to calibrate a linear model of the form shown here: $$Y\sim\beta_0+\beta_1X_1+\beta_2X_2$$ where the $\beta$ terms are the coefficients and the $X$ terms are the independent variables. ...
user avatar
  • 176
1 vote
0 answers
29 views

Least-squares fit of explicit parabolic sheet to scattered data points [duplicate]

For a given set of data points $$\{(x_i, y_i, z_i)\}$$ there exists some $$f_{ABC}(x,y)=Ax^2+Bxy+Cy^2$$ that minimizes $$\sum_i(f_{ABC}(x_i,y_i)-z_i)^2$$ $A$, $B$, and $C$ can be found quickly ...
user avatar
  • 365
6 votes
2 answers
2k views

Monotonic splines in Python [closed]

I am trying to find a procedure to fit data monotonically in Python. The data won’t be necessarily monotonic. I just would like to achieve a monotonic fit because of theoretical assumptions. I ...
user avatar
  • 71
4 votes
1 answer
393 views

How do you apply constrains on parameters in Bayesian modeling?

In Frequentist modeling, I know how to fit and interpret models with constraints on the parameter space. Let's say I'm fitting a $N(\mu, \sigma^2)$ distribution to my data $x_1, \dots, x_n$, and I ...
user avatar
  • 2,089
4 votes
0 answers
65 views

Linear regression of higher order polynomial with slope constraint

I am trying to constrain the coefficients on a higher order polynomial (let's say an order 6) for the curve to be decreasing. I have found this link, where the fitting of a 3rd order polynomial is ...
user avatar
3 votes
1 answer
69 views

How to fit several linear models with linear intercept-slope constrain?

I have a data where several groups exist: $i = 1, 2, 3...n$ ($n$ about 600 groups in total). For group $i$ theory predicts a linear relation $y = a_i + b_i * x$ with $a_i$ and $b_i$ intercept and ...
user avatar
  • 33
2 votes
2 answers
575 views

Adding limits to regression coefficients

For my problem, I have data that contains daily observations of the total time and the volumes of task A completed, task B completed, C, D.. and I am looking to estimate the time it takes to solve ...
user avatar
0 votes
0 answers
119 views

Is there a conjugate prior for normal location family N(x|u,1) such that the mean is always positive?

The conjugate prior to normal location family is usually a normal distribution. However, I want to constrain the mean to be positive. Is there a conjugate prior to the normal location family $x\sim N(...
user avatar
3 votes
1 answer
111 views

Model/link function to deal with dependent variable in range [-1,1]?

My dependent variable, $Y$, contains values anywhere from -1 to 1 (i.e. it is bounded continuously on the range $[-1,1]$). I know that a regular OLS regression on such a variable would sometimes ...
user avatar
1 vote
0 answers
106 views

Sum of residuals in constrained linear regression

Does sum of residuals == 0 hold even if we add ordinality constraints for the coefficients in a regression model? It would be great if someone can point out to any literature (paper/presentation/...
user avatar
0 votes
0 answers
172 views

Closed-form solutions for constrained multiple linear regression

Normally a multiple linear regression is unconstrained $$y=X\beta+\epsilon$$ so that closed-form solutions in the case of data orthogonality ($X^\top X=I$) are $$\beta=(X^\top X)^{-1} X^\top y$$ ...
user avatar
  • 3,139
0 votes
0 answers
269 views

Restricted Weighted Linear Regression in R

I have to follwing issue. I would like to run a linear regression imposing a constraint on the weighted coefficients. Let me construct an example: Consider the following cross-sectional regression $...
user avatar
2 votes
2 answers
1k views

Predict a vector of values with constraints? [duplicate]

I am aware of a variety of methods for simultaneously predicting multiple outcomes known sometimes as multivariate regression/analysis. However, my situation is a little more special. I am trying to ...
user avatar
0 votes
1 answer
320 views

How can you constrain values to be positive when fitting a model?

I'm currently fitting a model using maximum likelihood estimation on biological data (electroencephalography). Basically, I'm fitting normal distributions to several subsets of data (experimental ...
user avatar
4 votes
2 answers
418 views

Equality-constrained least-squares when the matrix is singular

I have to solve the linear system $Ax = b$ in the least-square sense, where matrix $A$ is singular. To resolve this, I introduce $Cx = d$. However, even after this, I am not able to solve it using <...
user avatar
  • 151
2 votes
1 answer
2k views

Is it possible to apply a monotonicity constraint on a Gaussian process regression fit?

Below is a code using scikit-learn where I simply apply Gaussian process regression (GPR) on a set of observed data to produce an expected fit. I know physically that this curve should be ...
user avatar
  • 447
1 vote
0 answers
701 views

R - Linear Regression with parameters constraint - contr.sum contrast

I am facing a problem with running a simple OLS regression with two categorical independent variable. I would like to impose that the sum of the coefficients referred to one variable (x in the ...
user avatar
3 votes
0 answers
105 views

How to run linear regression with constraints in R? [duplicate]

If I have the following data n<-1000 x1<-rnorm(n,1,1) x2<-rnorm(n,2,2) x3<-rnorm(n,3,3) e<-rnorm(n) y<-3+0.5*x1+0.2*x2+0.3*x3+e I want to fit a ...
user avatar
  • 181
2 votes
1 answer
1k views

method for predicting a curve

I have data on several curves. the data is of the form: curve_id x y and there are many x/y pairs for each curve and x is limited to some known range. overall, the curves look quite similar in ...
user avatar
  • 551
0 votes
1 answer
575 views

polynomial regression in R: how to add hard constraints (go through specified points)? [duplicate]

I'd like to perform polynomial regression on my data (which lies in [0,10]), but I need to ensure that the endpoints of the range are fixed, i.e. that the curve goes through (0,0) and (10,10). So the ...
user avatar
1 vote
0 answers
294 views

nlme ignoring certain control arguments to nlminb

Issue: Do constrained optimization of parameters in nlme::nlme I'm trying to fit a non-linear mixed effects model using nlme::nlme, which can use 2 optimization schemes: stats::nlminb or stats::nlm. ...
user avatar
  • 31
0 votes
1 answer
77 views

Confidence limits for constrained penalized log likelihood model

I am estimating parameter $\beta$ as: \begin{align} \hat \beta &= \mathop{\mathrm{arg\,max}}_\beta \;\; l(\beta;X,y) - \frac{\lambda}{2}\left(\tilde y-g(\beta,\tilde X)\right)^\prime C^\prime C\...
user avatar
  • 113
4 votes
1 answer
612 views

Is constrained (nonnegative) least squares a form a regularisation?

Is nonnegative least squares already a form of regularisation? By adding a constraint that $\beta \geq 0$ (the coefficients), does it make sense to add another regularisation term as in LASSO or ridge ...
user avatar
  • 8,025