1
$\begingroup$

Let's say we have an independent variable X such that

$X=\sum_i^n x_i$

and then we have a covariate matrix $C$

So we will have the following regression model:

$Y=X\beta_1+C\beta_2+\epsilon$

However, what we really want to do is to find $k$ such that for $X_k=\sum_i^k x_i$

we can get the highest r-square out from $Y=X_k\beta_1+C\beta_2+\epsilon$

Currently we perform a sequential test such we calculate the r-square for $X_1, X_2,...,X_n$ and then identify the model with the highest r-square.

Are there any better method for this selection process?

Edit: Thank you all for your comments First, the reason why we use $X=\sum_i^n x_i$ instead of individual $x_i$ in our model is that there can be a huge number of $x_i$. So in some situation, it might not be possible for us to solve the equation with that many $x_i$.

Here, in our settings, $X$ is considered as an aggregated score where we have the basic assumption that the amount of information in $x_1>x_2>...>x_n$, where we assume for any $x_i$ from $k+1$ to $n$ contains mostly noise.

From what we understand at the moment, our method of finding the largest r-square is less than optimum and might lead to overfitting. However, it is the only thing that we can figure out at the moment. Therefore if there is any analytical way for us to find $k$ for the best predictive power, it might benefit us tremendously

$\endgroup$
6
  • 1
    $\begingroup$ Why do you want to use $X_k = \sum^k_{i=1} X_i$ rather then individual $X_i$'s? $\endgroup$
    – Tim
    Sep 21, 2016 at 11:31
  • 2
    $\begingroup$ This type of selection of predictors, based on r-squared values from a particular data sample, has many problems. See the feature-selection tag on this site. Providing more information on what you are trying to accomplish with your model might lead to suggestions for better approaches. $\endgroup$
    – EdM
    Sep 21, 2016 at 14:23
  • $\begingroup$ Is the indexing of the $x_i$ meaningful or are there some (assumed) relationships among the $x_i$? If not, it might be of interest to observe that your question amounts to a limited form of model selection process in which the variables $C$ will be included and exactly one of the $\{X_1,X_2,\ldots,X_n\}$ will be included with them. By first regressing $Y$ and each of the $X_k$ on $C$ and retaining the residuals, this is equivalent to the same situation with $C$ removed: in other words, you seek the best one out of $k$ possible variables to model $Y=\beta X_k+\epsilon$. $\endgroup$
    – whuber
    Sep 21, 2016 at 14:36
  • $\begingroup$ NB: your test does not appear to be a "sequential test" as commonly understood. It's a search of all possible solutions. A sequential test would take the $X_i$ in the order given and apply some criterion to determine when to stop testing. It therefore might not ever consider including $X_n$, for instance, if it stopped early. $\endgroup$
    – whuber
    Sep 21, 2016 at 14:38
  • $\begingroup$ When you say "there can be a huge number of $x_i$," do you mean that different observations can have different values of $n$, or something else (perhaps, different observations have different numbers of non-zero values of $x_i$)? And approximately how huge is "huge" relative to the numbers of observations? $\endgroup$
    – EdM
    Sep 22, 2016 at 16:36

1 Answer 1

1
$\begingroup$

With a set of $x_i$ ordered according to their assumed amount of information, you might be able to get what you need with the LASSO, using the $x_i$ rather than their running sums $X_k$ as the candidate predictor variables. This is a standard approach when the number of candidate predictors greatly exceeds the number of observations. It's explained, for example, in Chapter 6 of An Introduction to Statistical Learning, with a worked exercise using the glmnet function from the package of the same name in R.

With large numbers of predictors LASSO sets coefficients of most to zero, with the number of retained coefficients determined by a penalty value. The penalty value is typically determined by cross-validation, for example choosing the penalty that minimizes cross-validated prediction error. There is, however, no "analytic" way to determine the number of retained predictors.

If the $x_i$ are more or less in the same order of assumed importance among all your observations, then the LASSO should maintain only the most important $x_i$ in the model. The coefficients of the individual $x_i$ will generally differ, so this is not the same as the running sums $X_k$ as you are presently using. If you need to use running sums instead of individual $x_i$ there might be a way to force all the maintained $x_i$ to have the same coefficient (and thus effectively use a sum of maintained $x_i$ as the predictor), but that's outside my experience or knowledge.

If you do try LASSO be careful about how it's implemented. Usually the predictor variables are pre-standardized so that they all are considered in comparable scales. Consider whether that's what you want here (standardize argument to glmnet). Also, it seems that you want to maintain all of the predictors in your covariate matrix $C$. If so, you can work with the residuals after accounting for $C$, as suggested in a comment from @whuber, or specify that the variables in $C$ are maintained without penalty (penalty.factor argument to glmnet).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.