I am attempting to explain a dichotomous outcome variable using a large set of continuous valued sensor-derived variables. Many of these variables are highly mutually correlated, some are based on solid physical concepts (and therefore easier to interpret) others are more abstract and difficult to explain.

I am attempting to examine the association of these sensor variables with the outcome (i.e. which variables or combination of variables can best explain the dichotomous outcome variable?) a secondary objective is to determine if a subset of these variables can be used to explain the dichotomous outcome.

I wish to avoid using a stepwise fitting procedure (due to the danger of overfitting and, a feeling that the more easily interpretable variables should be given preference in the model over highly correlated but less easily interpretable variables). In short, I am looking for the true associations rather than noisy surrogates using correlated but less globally informative variables.

To avoid multicollinearity in the analyses, I reduced the number of variables using logistic regression by block analysis. Sensor derived variables were grouped by type into blocks. The dichotomized outcome variable was used as the dependent variable in each sub-group. Working with each block, I performed a logistic regression on each independent variable and only those which were significant (α < 0.05) were retained in each block. Through this procedure I excluded all non significant variables from the analyses for the final model. I then generated a final logistic regression model using the results of each sub-group analysis.

I would great appreciate opinions as to whether this is a valid approach? Can the odds ratios from the final logistic regression model can be used to interpret the associations of the included sensor variables with the outcome variable in the manner of a hypothesis test?


1 Answer 1


Your approach seems valid for me. However, it does not consider conditional or interdependent relationship of variables within every group. Your are building a regression model for every individual feature rather than considering a specific combination. In simple terms, a variable that is not significant by itself can be significant when considered with another feature.

Generally, there are many sound methods for feature selection including filter and wrapper models. You may use some ranking method such as mRMR to rank your variables. Then, take the top k ranked features as well as the ones you deem relevant to include.

Regarding the wrapper model, you may optimize the performance of your regression models using any search method such as GA. For preserving the selection of the admired featuers, you can add constraints to the optimization process. This way will lead to selection of variables that optimize the performance given some other featuers fixed as intrepretable ones.

To interpret the association or to measure "relevance" of any selected variable, various metrics can be applied based on your definition of relevance (there is no single agreed upon definition of relevance). Once again, you may use any correlation or mutual information based metric e.g. mRMR, JMI and ICAP.

  • $\begingroup$ thanks. I had actually simplified what I was doing as I had also added interaction terms to the model. I have used wrapper based feature selection approaches with constraints in the past but often the models produced are actually very difficult to interpret (hence this simpler logistic approach) $\endgroup$
    – BGreene
    Feb 28, 2013 at 10:49
  • $\begingroup$ Would you elaborate more why the resulted logistic regression models based on the wrapper approach were very difficult to interpret compared to the simpler version in your post? $\endgroup$
    – soufanom
    Mar 1, 2013 at 11:16
  • $\begingroup$ Sure, I was trying to explain the relationship between a number of sensor based parameters and a clinical outcome. Of primary interest is the degree of association with the outcome variable. A logistic regression model is easier to explain from a real world clinical perspective as it provides clinically interstate coefficients (odds ratios). $\endgroup$
    – BGreene
    Mar 2, 2013 at 12:24
  • $\begingroup$ Still, I do not see why the wrapper approach was not useful. The learner in the wrapper model is just a black box to the search method. Thus, we may wrap genetic algorithm around the logistic regression itself and search for the best feature subset. Why then, using a wrapper around logistic regression is worse than the logistic regression by itself? $\endgroup$
    – soufanom
    Mar 3, 2013 at 7:16
  • $\begingroup$ Again, I was not trying to build an optimal model for prediction, I was using the model to explain the effect of the predictors on the outcome variable, i.e. an explanatory rather than predictive regression model. $\endgroup$
    – BGreene
    Mar 4, 2013 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.