Exploratory analysis: regression model with mutually correlated predictors to explain a dichotomous outcome?

Question

I am attempting to explain a dichotomous outcome variable using a large set of continuous valued sensor-derived variables. Many of these variables are highly mutually correlated, some are based on solid physical concepts (and therefore easier to interpret) others are more abstract and difficult to explain.

I am attempting to examine the association of these sensor variables with the outcome (i.e. which variables or combination of variables can best explain the dichotomous outcome variable?) a secondary objective is to determine if a subset of these variables can be used to explain the dichotomous outcome.

I wish to avoid using a stepwise fitting procedure (due to the danger of overfitting and, a feeling that the more easily interpretable variables should be given preference in the model over highly correlated but less easily interpretable variables). In short, I am looking for the true associations rather than noisy surrogates using correlated but less globally informative variables.

To avoid multicollinearity in the analyses, I reduced the number of variables using logistic regression by block analysis. Sensor derived variables were grouped by type into blocks. The dichotomized outcome variable was used as the dependent variable in each sub-group. Working with each block, I performed a logistic regression on each independent variable and only those which were significant (α < 0.05) were retained in each block. Through this procedure I excluded all non significant variables from the analyses for the final model. I then generated a final logistic regression model using the results of each sub-group analysis.

I would great appreciate opinions as to whether this is a valid approach? Can the odds ratios from the final logistic regression model can be used to interpret the associations of the included sensor variables with the outcome variable in the manner of a hypothesis test?

Answer 1

Your approach seems valid for me. However, it does not consider conditional or interdependent relationship of variables within every group. Your are building a regression model for every individual feature rather than considering a specific combination. In simple terms, a variable that is not significant by itself can be significant when considered with another feature.

Generally, there are many sound methods for feature selection including filter and wrapper models. You may use some ranking method such as mRMR to rank your variables. Then, take the top k ranked features as well as the ones you deem relevant to include.

Regarding the wrapper model, you may optimize the performance of your regression models using any search method such as GA. For preserving the selection of the admired featuers, you can add constraints to the optimization process. This way will lead to selection of variables that optimize the performance given some other featuers fixed as intrepretable ones.

To interpret the association or to measure "relevance" of any selected variable, various metrics can be applied based on your definition of relevance (there is no single agreed upon definition of relevance). Once again, you may use any correlation or mutual information based metric e.g. mRMR, JMI and ICAP.