I have a kind of obscure question about using the Least Angle Regression (LARS) algorithm for variable selection. If I'm understanding it right, my professor formulates LARS as such:

$$\mathbb{min}\ \hat{y}^T\hat{y}\ \mathbb{such}\ \mathbb{that}\ |t_j| \le t\ \forall\ j$$ where $t_j\ = (x_j^Tx_j)^{-1/2}x_j^T(y-\hat{y})/\sigma$.

The dataset I'm working with deals with grades for a small class ($\mathbb{n = 22}$ students, $\mathbb{p = 16}$ graded assignments) and the response is the students' grade on the final exam. I stumbled across a result that's either mildly interesting or a coincidence, and was wondering if anybody knows about something mathematically similar to this that's related to variable selection?

I standardized the predictors and centered the response and ran a Leave-One-Out Cross Validation for each of the $\mathbb{p}$ steps in the LARS algorithm. I wanted to figure out a way to tell which step I should stop at.

One thing I thought of was to create a $\hat{y_i}^T\hat{y_i}$ vector with $\mathbb{n}$ elements where $\hat{y_i}$ is the predicted model with the $\mathbb{i^{th}}$ observation missing. $\hat{y}^T\hat{y}$ is the value of the full predicted model at each step. From there I made $\mathbb{p}$ "z-scores" calculated so that: $$z_j= \dfrac{\hat{y}^T\hat{y}-mean(\hat{y_i}^T\hat{y_i})} {sd(\hat{y_i}^T\hat{y_i})}$$ What I found was that $\mathbb{z_j}$ decreased until it reached a minimum at $\mathbb{z_8}$ and then increased. It so happened that the model with 8 variables was the exact same model we reached when we decided to only include positive $\mathbb{\beta_j} \mathbb{'s}$ ($\beta_9$ was negative) since that made realistic sense. I also ran this data through normal stepwise regression and got something like a minimum AIC at 3 variables and minimum BIC at 13 variables.

I'm trying to make my code for this technically correct and run this with different data, but my long winded question is essentially if anyone thinks that this something mathematically related to an idea from regression/variable selection or just a strange uninteresting coincidence?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.