4
$\begingroup$

What variable selection approach should I consider if I have thousands of predictors with clusters that are extremely correlated?

For example I might have a predictor set $X:= \{A_1,A_2,A_3,A_4,...,A_{39},B_1,B_2,...,B_{44},C_1,C_2,...\}$ with cardinality $|X| > 2000$. Consider the case where all $\rho(A_i,A_j)$ are very high, and similarly for $B$, $C$, ....

Correlated predictors aren't correlated "naturally"; it's a result of the feature engineering process. This is because all $A_i$ are hand engineered from the same underlying data with small variations in hand-engineering methodology, e.g. I use a thinner pass band on $A_2$ than I did for $A_1$ in my denoising approach but everything else is the same.

My goal is to improve out of sample accuracy in my classification model.

One approach would just be to try everything: non-negative garotte, ridge, lasso, elastic nets, random subspace learning, PCA/manifold learning, least angle regression and pick the one that's best in my out of sample dataset. But specific methods that are good at dealing with the above would be appreciated.

Note that my out of sample data is extensive in terms of sample size.

Improve this question
$\endgroup$
1
  • $\begingroup$ I am also working with correalted data and i am getting quite good results with lasso. $\endgroup$ – user44277 Apr 22 '14 at 19:34
1
$\begingroup$

I would do the forward stepwise selection, adding predictors as long as the correlation with residuals is significant, and then do some regularization (ridge, lasso, elastic nets). There are 2-3 metaparameters: forward stepwise termination constraint, and 1 or 2 regularization parameters. These metaparameters are determined via cross-validation.

If you want to take into account non-linearity, you could try random forest, which produces good results when there are many predictors. But it is slow.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ My data is not of the form to trust a test on the Pearson correlation. In fact it would violate almost every assumption of such a test. Is this still the approach you would take? $\endgroup$ – user2763361 Oct 11 '13 at 15:37
  • $\begingroup$ How did you produce the data? Why you cannot trust a test on the Pearson correlation? Please elaborate. $\endgroup$ – user31264 Oct 11 '13 at 15:42
  • $\begingroup$ Heteroscedasticity, non-normality, autocorrelation, non stationarity. Pearson correlation test unbiasedness is known to be extremely susceptible to violations of its assumptions. $\endgroup$ – user2763361 Oct 11 '13 at 15:51
  • $\begingroup$ What do you mean by autocorrelation and non-stationarity? Your dependent variable and predictors are time series? $\endgroup$ – user31264 Oct 12 '13 at 6:04
  • $\begingroup$ Yes these are time series which display autocorrelation and non-stationarity. e.g. in the post I talk about pass bands and denoising. $\endgroup$ – user2763361 Oct 12 '13 at 8:14
1
$\begingroup$

Features extracted from image/signal processing tend to get correlated a lot! This is not a very bad thing if you have enough samples.

From my experience, a classifier with small variance tend to work well (ex. logistic regression). They have less chances of overfitting the train data.

Another idea that I employed is the Additive logistic regression here and here some references. They are already implemented in Weka. They are slower than the logistic models. In the same time they have the great advantage that they perform a feature selection while learning. Moreover, the model is human friendly so you can see what features are more relevant.

Hope it helps

Improve this answer
$\endgroup$
2
  • $\begingroup$ I think the question is about regression, not classification. $\endgroup$ – steveo'america Jun 27 '17 at 17:53
  • $\begingroup$ Different faces of the same coin. $\endgroup$ – visoft Jul 3 '17 at 8:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.