2
$\begingroup$

My dependent variable is continuous variable that measures the (potential) success of a person in some activity. I have hundreds of binary indicators, each indicates about the existence of a specific beneficial behavior or property (e.g. the person typically wakes up before 8am). each of these indicators is positively linked to the outcome both theoretically and empirically (although some of the empirical differences are insignificant). Of course that the indicators are correlated.

My task is to find a subset of 10-20 indicators that best predict the outcome. Ideally, what I want to have is a subset of indicators from which I'll construct a single variable - the number of positive indicators from that subset for a person, and wish that this variable has the highest possible correlation with the outcome.

What I did first is a linear regression with all the indicators in. It gave me lots of negative coefficients (which makes sense mathematically, but is not intuitive as you'd expect that each variable will contribute something), and it didn't help much in choosing the subset.

I have also tried a sort of a genetic algorithm optimization to help me choose such subset. It helped, but I feel I'm missing some statistical way of doing that (PCA perhaps?), and would be happy to get any leads/references/suggestions about approaching this problem, which looks like a model selection problem.

Any ideas?

Thanks.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

This should be a comment, but I'm still trying to work out how to do that on this site.

If you've tried genetic algorithms, I assume you've tried a lot of other options. Might be helpful listing them. It sounds from the question, but not clear, that you have (1) a p>n problem (2) collinearity problem (3) are ignoring interactions.

Based on that elastic-net (glmnet package) might be useful. You want a subset selection method, but given p>n you'd prefer a shrinkage method, and given collinearity elastic-net might be better than lasso.

Dimension reduction methods (like PCA) usually still use all the variables to generate the principal components - so doesn't seem that useful to you.

I wasn't that clear about the arbitrary cut-off of 10-20 variables. If you do just want 10-20 variables and that's it, you could rank predictive importance and then choose the top 10-20. This can be done after many learning models - but think easiest after boosted regression.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for your comment. N>P in my case. the reason I'm looking for a small subset is mundane one. People need to use the model. they can handle monitoring only a limited number of indicators. I am ignoring interactions just because I haven't seen a lot of them in the data (I checked). The correlations among indicators are not very big, so for small sets of indicators I don't expect a lot of multicollinearity, but for large sets it might be a problem. This is also a good reason for looking for small subsets. $\endgroup$
    – amit
    Nov 12, 2013 at 9:33
  • $\begingroup$ I actually haven't looked into lots of methods. I used GA simply because as an algorithm developer I did such stuff before. $\endgroup$
    – amit
    Nov 12, 2013 at 9:34
  • $\begingroup$ Hope I didn't waste time then. Not sure if I have much useful to add. I think if N>P you could use any feature selection method you wanted to (from stepwise to lasso) and just choose what is easiest to implement. It shouldn't be an issue. (1) these methods may include weak predictors that are statistically significant. but since most predictors are binary you can compare coefficients and include strongest predictors (2) leap-bound (cran.r-project.org/web/packages/leaps/leaps.pdf) is a popular method for this sort of problem. But I haven't used it myself. $\endgroup$
    – charles
    Nov 12, 2013 at 18:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.