# Regression by multiple dependent variables with constraints & feature selection

I have a data set of 1000 records. Each record has three dependent variables $y_1, y_2, y_3$ and 100 independent variables $x_1,...,x_{100}$, where the dependent variable $y_i$ satisfies:

1. $0\le y_i \le1$
2. $y_1 + y_2 + y_3 =1$

I.e. $y_i$ represent the probability of a observation belonging to one of the three classes.

Q1: How can I build a (multivariate linear) model using $x_1,...,x_{100}$ to predict ($y_1,y_2,y_3$)? Is any R package available? How can I implement it?

Q2: Since there are so many independent variables $x_1,...,x_{100}$ (features), is it possible to do feature selection using LASSO, SCAD, or elastic net for this multivariate linear model using tools in glmnet package?

• Q1: keep me updated if you got an ansver. Q2: you may run PCA first. – user49432 Jul 3 '14 at 11:46
• What you are asking about is multinomial logistic regression in R. You can do this with the multinom() function in the nnet CRAN package. Have a look at here. – Zhubarb Jul 3 '14 at 12:49
• @Berkmeister the $y_i$s are continuous and therefore logistic regression doesn't work here. – pes Mar 14 '17 at 16:41

You can try the PLS regression which is fit for datasets with multiple dependant variables.

Q1 : For a multivariate linear model, combine it with the clusterwise approach.

Q2 : PLS is built to work with many independant variables, even with more than observations.

Here there is a R package available.

We're working on another open source implementation which will improve it, it will be available soon.

Q1: I'd think a bit harder about exactly how you might want to predict $y_1, y_2, y_3$. If you're only interested in the one with the highest value, you might re-code the independent variables into a single outcome column, where the value is Group 1, Group 2, or Group 3, depending on which of your $y$ is highest in each observations. Some pseudo-code:

if max($y_1,y_2,y_3$) = $y_1$, then Outcome = Group 1...

Afterwards, you can use any multivariate linear model and task it on fitting a model to your new outcome variable.

Q2: You have two major options: feature selection and signal extraction. I'm a fan of feature selection with random forests, but many methods are capable of generating variable importances with supervised and unsupervised approaches. For signal extraction, you might use PLS, PCA, ICA, etc.