Regression analysis generating predictors from multiple measures

Question

My setting is the following, in my dataset, I have different measures (14 types of measures), let's call them: $m_1, m_2, \ldots, m_{14}$. I'd like to find the regression with the maximum $r^2$, using as predictors, the sum of these measures, for instance:

predictor 1: $p_1=w_1 + w_2$

predictor 2: $p_2=w_1 + w_3$

$\ldots$

predictor X: $p_x = w_1 + w_2 + w_3 + w_4$

etc.

My problem is combinatorics, I can create up to 16383 predictors (which are different combinations of the sum of measures), and thus, a brute force approach is intractable. Is there a better solution for this? Is this a known problem?

Answer 1

This is a variation on the known problem of variable selection in regression. In the general case, you have $m$ potential explanatory variables, which yields $2^m$ possible sets of variables that could go into your model (including the case with no variables, and the case with all of them). The all-possible-model method fits all of these $2^m$ possible regression models (for a given functional form). In cases where this is computationally infeasible, it is usual to fall back on stepwise variable selection methods (e.g., forward selection, backward elimination, or bidirectional elimination).

Now, your case differs slightly from the normal situation, insofar as you are putting your explanatory variables in only through a single sum, rather than as separate explanatory variables. Nevertheless, the problem is completely analogous, and you can use exactly the same methods. So, for example, if you were to use backward elimination, you would start by comparing the model with all 14 variables in the sum to each of the 14 models with 13 variables in the sum, which would allow you to eliminate one (or keep all of them and halt the selection). Then you would take your new model with 13 variables in the sum and compare this to each of the 12 models with only 12 of those variables in the sum, and so on. This should lead you to a reasonable model.

It is important to note that there is no guarantee that you will find the model that maximises the $R^2$ unless you use the all-possible-models selection method. Stepwise selection methods are usually reasonable, but sometimes they find this set of variables, and sometimes they don't. Good luck.

Answer 2

Not a full answer, but an approach:

If I'm interpretting your question correctly, you want to find the $p_i$ that has the highest linear correlation with your outcome $Y$, where $p_i$ is a sum of a subset of $w$.

We can reframe this as a multiple regression problem of $Y$ on the $w_i$ directly, with the constraint that each coefficient $\beta_i$ is either $0$ or equal to all the other $\beta_i$ - we could write this as

$\min \sum (Y - w\beta)^2 $ such that $ \beta_i * \beta_j * (\beta_i - \beta_j) = 0$ for each pair of coefficients.

I think this is something a nonlinear optimization package (such as IPOpt) could handle, or you could try to work it out analytically using Langrange multipliers if you're so inclined.