I'm running a regression analysis with independent variables $X_{1}, X_{2}, \cdots, X_{n}$ and dependent variable $Y$. There is a constraint among some of the independent variables, say, $X_{1} + X_{2} + X_{3} = 1$. What kinds of regression models (or other data science techniques) could be used in this scenario?
6
-
$\begingroup$ Maybe compositional data. $\endgroup$– user2974951Commented Sep 2, 2019 at 12:16
-
$\begingroup$ With just one dependent variable, this is multiple rather than multivariate. No one has yet given the simplest response which is that if some of your predictors add to a total, then you can dispense with one of them. Anything more complicated is not essential on that ground alone. To see this, imagine a two-category classification yielding say fraction which are red and fraction which are not red. There is no more information in not red than in red, so choose one of the two variables. $\endgroup$– Nick CoxCommented Sep 2, 2019 at 13:18
-
1$\begingroup$ @NickCox ok I think I got your point. But here to me, if you wish to interpret the constraint on predictors alone (not the predictors time the beta), it does not mean that for every sample you have the constraint holding, so that you can get rid of one predictor in the whole model. In my personal opinion (which we can debate clearly) it rather means that you regress iff the selected predictors satisfy that relationship. So you just consider the samples where such relationship holds. Is it correct Nick? Or maybe Am I missing your point? $\endgroup$– Fr1Commented Sep 2, 2019 at 13:30
-
$\begingroup$ Thank you @NickCox, I changed the title to multiple regression. $\endgroup$– waynelee1217Commented Sep 2, 2019 at 13:42
-
1$\begingroup$ 'As @user2974951 hints briefly. data now often called compositional are the fractions (proportions, percents, whatever) of mutually exclusive categories, so constrained in principle to add to 1 (100%). . Examples that are common are: texture of materials; chemical composition of materials; categories of expenditure; etc. $\endgroup$– Nick CoxCommented Sep 2, 2019 at 13:59
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
12
Maybe the constraint is on coefficients of such independent variables, or the sum of the products $x_{1} \beta_{1} + x_{2} \beta_{2} + x_{3} \beta_{3} $?
If this is the case you can use the Constrained OLS (look at this) which is a minimization of the sum of squared residuals subject to a constraint or set of constraints solved through the Lagrangian. So it is the constrained-optimum version of the typical unconstrained minimization of squared residuals of OLS. Notice that the principle of optimizing a cost function or maximizing a target function subject to a constraint be extended outside OLS estimators. For example you can perform Constrained Maximum Likelihood like this and this.
-
$\begingroup$ Question seems clear to me. OP says that the constraint is on the values of the predictors, not their coefficients or the predictions. $\endgroup$– Nick CoxCommented Sep 2, 2019 at 13:17
-
1$\begingroup$ In that case I am eager to know the answer as well. I never saw one, which does mean it does not exist. But from the logical standpoint what are you doing here? Are you regressing y on the row x only when the row x meets the constraint? It may be.. however it is not very usual, For this reason I interpreted the question as the far more common case where you put a constraint on the beta or fitted values. Did you hear of it Nick? $\endgroup$– Fr1Commented Sep 2, 2019 at 13:21
-
$\begingroup$ Thank you @Fr1. I'm running a dataset with each row meets the constraint I mentioned above. I believe I'm not constraining $\beta_{1}X_{1}+\beta_{2}X_{2}+\beta_{3}X_{3}=1$. Or should I have no constraint and just interpret the values of $\beta_{1}, \beta_{2}, \beta_{3}$? $\endgroup$ Commented Sep 2, 2019 at 13:47
-
1$\begingroup$ So.. if your desired constraint is as you said initially X1+...+Xn=1 and you are saying that in your dataset each row is meeting such constraint (right?), then I believe that your regression will be a common OLS using that dataset (as constraints are already satisfied by all rows). If this is the case for your data and your intentions, well in that case you must not impose the b1X1+...+bnXn constraint. Let’s hear Nick’s opinion. $\endgroup$– Fr1Commented Sep 2, 2019 at 13:52
-
1$\begingroup$ @NickCox no I was referring to “listening to your opinion” just to say “let’s wait for Nick comments because they are valuable”. Having said that, what you are saying is right as long as all the observations (I.e. all the samples for the multiple predictors) ALREADY satisfy the constraints. IF NOT, then you have to filter the dataset and consider the only rows where the constraint is satisfied. Do you agree? Then ONLY those rows will be included into the regression dataset. And clearly for THOSE rows, you can use one less predictor. Not for ALL initial rows. This is what I am saying. $\endgroup$– Fr1Commented Sep 2, 2019 at 14:09