How to choose factors in constrained Linear models? I was trying to do a linear model analysis where the parameters are constrained (sum to 1 and non-negative).But I found that is not obvious how to apply AIC function(or others) to the parameters selection as I wanted to use optimization to solve the linear problem.
Any idea or other suggestions?
thanks,
Wei
 A: OK, one can try the following procedure to get a AIC-like test.


*

*Solve the quadratic programming problem defined and get $x_{opt}$. Say k is the length of $x_{opt}$.

*Use $x_{opt}$ in a regular linear model; get fitted values and then calculate that model's residuals.

*Fit a Gaussian distribution to that residual vector. Usual implementation use an E-M algorithm but we will not expand on this here. Computationally one can use something like resDist = MASS::fitdistr(residVector, 'normal') in R.

*Use that Gaussian distribution to get the log-likelihood of the distribution object. Mathematically we would calculate the Gaussian log-likelihood as: $ \log(L(\theta)) =-\frac{|D|}{2}\log(2*\pi) -\frac{1}{2} \log(|K|) -\frac{1}{2}(y-\hat{y})^T K^{-1} (y-\hat{y})$, $K$ being the covariance structure of your model, $|D|$ being the number of points in your datasets, $\hat{y}$ the mean response and obviously $y$ being your dependent variable. Computationally one can use something like: myLL = logLik(resDist) in R.

*Use the number of parameters k from step 1 and the log-likelihood myLL from step 4 to get your AIC. (Or even better AICc so you somewhat correct for your finite sample.)


Caveats: 


*

*Between steps 2-3 we make the leap of faith that the residuals of this model are normally distributed. Are they? Maybe they are? You need to check that. Usually deviations from normality are not catastrophic but given the constraints you have you might be very far off. 

*The AIC you get is an upper bound for the AIC of that model. This is because your estimates for the parameters in $x_{opt}$ are not true maximum likelihood estimates. They would be only if your constrained and unconstrained solutions were equal. The fact that it is an upper bound means that it will only be good as an indicator. It is not a true AIC so when you report you should keep that in mind.

*On first instance do not bootstrap this model. Your $x_{opt}$ solution potentially lies close to (or even onto) your parameters' space boundary. As a consequence your bootstrap might be biased.

*Stepwise regression (because essentially this is what you want to do) is known to cause data-dredging which is a bad thing. It can lead to false interpretation of the results. Take any variable selection procedure with a grain of salt. This is issue is extesnsively discussed in Cross-Validated; eg. see the following two great threads for starters here and here.

