Skip to main content
Cross Validated

Return to Answer

replaced http://stats.stackexchange.com/ with https://stats.stackexchange.com/
Source Link
Community Bot
Community Bot
  • 1

Note that a logistic regression would also answer the question as originally posed. Each predictor variable in the logistic regression model, whether continuous or categorical, would have an associated regression coefficient and standard error expressed in comparable terms. The largest individual Wald statistics, the square of the ratio of each coefficient to its standard error, would indicate the variables with the "biggest" relations to Group membership in one well-defined way, as considered in the related context of survival analysis on this Cross Validated pagethis Cross Validated page. And the logistic regression would have the advantage of determining this relative importance for each variable with all the other variables taken into account.

For this application of logistic regression, with a large number of predictor variables, you will have to take some care to avoid perfect separation based on combinations of variablesperfect separation based on combinations of variables, particularly since you are interested in the ambiguous cases rather than in perfecting prediction of Group membership. Some ways to deal with this problem are discussed on this Cross Validated pagethis Cross Validated page and its links, and books like Regression Modeling Strategies provide approaches for reducing the set of predictors.

Note that a logistic regression would also answer the question as originally posed. Each predictor variable in the logistic regression model, whether continuous or categorical, would have an associated regression coefficient and standard error expressed in comparable terms. The largest individual Wald statistics, the square of the ratio of each coefficient to its standard error, would indicate the variables with the "biggest" relations to Group membership in one well-defined way, as considered in the related context of survival analysis on this Cross Validated page. And the logistic regression would have the advantage of determining this relative importance for each variable with all the other variables taken into account.

For this application of logistic regression, with a large number of predictor variables, you will have to take some care to avoid perfect separation based on combinations of variables, particularly since you are interested in the ambiguous cases rather than in perfecting prediction of Group membership. Some ways to deal with this problem are discussed on this Cross Validated page and its links, and books like Regression Modeling Strategies provide approaches for reducing the set of predictors.

Note that a logistic regression would also answer the question as originally posed. Each predictor variable in the logistic regression model, whether continuous or categorical, would have an associated regression coefficient and standard error expressed in comparable terms. The largest individual Wald statistics, the square of the ratio of each coefficient to its standard error, would indicate the variables with the "biggest" relations to Group membership in one well-defined way, as considered in the related context of survival analysis on this Cross Validated page. And the logistic regression would have the advantage of determining this relative importance for each variable with all the other variables taken into account.

For this application of logistic regression, with a large number of predictor variables, you will have to take some care to avoid perfect separation based on combinations of variables, particularly since you are interested in the ambiguous cases rather than in perfecting prediction of Group membership. Some ways to deal with this problem are discussed on this Cross Validated page and its links, and books like Regression Modeling Strategies provide approaches for reducing the set of predictors.

Bounty Ended with 50 reputation awarded by Scortchi♦
specified formulas for log odds ratio and its standard error
Source Link
EdM
EdM
  • 101.4k
  • 11
  • 102
  • 303

For continuous variables, you proposed a Z-test statistic (ratio of difference between group means to the standard errorstandard error of that difference) as a measure of how "big" the effect is. This defines "big" in terms of your ability to detect athe magnitude of difference between the two Groups (givenrelative to the variability anderror in the numbersestimate of cases), which makes sensethe difference. That's a perfectly reasonable definition.

For 2 x 2 categorical comparisons, the log odds ratio and its standard errorstandard error (as might come from logistic regression) can similarly be used to form a Z-test statistic, as they also take a normal distribution in the limit. These are easily calculated from a contingency table. If $A,B$ represent the Groups; 1,2 represent the two classes of a categorical variable; and $N$ represents the number of cases with the indicated combination of Group and class, then the log odds ratio is:

$$\log\left(\frac{N_{A1}N_{B2}}{N_{A2}N_{B1}}\right)$$

with approximate standard error:

$$\sqrt{\dfrac{1}{N_{A1}} + \dfrac{1}{N_{A2}} + \dfrac{1}{N_{B1}} + \dfrac{1}{N_{B2}}}$$

If any of your putative predictor variables are categorical with 3 or more levels, the multinomial regression of that variable against Group can be set up as a set of independent binary comparisons to get similar statistics.

That way you have comparable measures of "big" for continuous and categorical variables.

For continuous variables, you proposed a Z-test statistic (ratio of difference between group means to the standard error of that difference) as a measure of how "big" the effect is. This defines "big" in terms of your ability to detect a difference between the two Groups (given the variability and the numbers of cases), which makes sense.

For 2 x 2 categorical comparisons, the log odds ratio and its standard error (as might come from logistic regression) can similarly be used to form a Z-test statistic, as they also take a normal distribution in the limit. These are easily calculated from a contingency table. If any of your putative predictor variables are categorical with 3 or more levels, the multinomial regression of that variable against Group can be set up as a set of independent binary comparisons to get similar statistics.

For continuous variables, you proposed a Z-test statistic (ratio of difference between group means to the standard error of that difference) as a measure of how "big" the effect is. This defines "big" in terms of the magnitude of difference between the two Groups relative to the error in the estimate of the difference. That's a perfectly reasonable definition.

For 2 x 2 categorical comparisons, the log odds ratio and its standard error (as might come from logistic regression) can similarly be used to form a Z-test statistic, as they also take a normal distribution in the limit. These are easily calculated from a contingency table. If $A,B$ represent the Groups; 1,2 represent the two classes of a categorical variable; and $N$ represents the number of cases with the indicated combination of Group and class, then the log odds ratio is:

$$\log\left(\frac{N_{A1}N_{B2}}{N_{A2}N_{B1}}\right)$$

with approximate standard error:

$$\sqrt{\dfrac{1}{N_{A1}} + \dfrac{1}{N_{A2}} + \dfrac{1}{N_{B1}} + \dfrac{1}{N_{B2}}}$$

If any of your putative predictor variables are categorical with 3 or more levels, the multinomial regression of that variable against Group can be set up as a set of independent binary comparisons to get similar statistics.

That way you have comparable measures of "big" for continuous and categorical variables.

Source Link
EdM
EdM
  • 101.4k
  • 11
  • 102
  • 303

Goal of project

According to your comment the ultimate goal is "to create a scoring formula to identify members of one population who most closely resemble the other population" (emphasis added). Let's call those the "ambiguous" cases, and call the populations "Group A" and "Group B."

Problems with proposed approach

Looking for the biggest univariate differences (however defined) between the Groups might not be what is needed to identify ambiguous cases.

First for "biggest": consider the extreme case of perfect separation, a logical variable that is TRUE for all members of Group A, but FALSE for all members of Group B. That would score best of all as a "biggest" difference, but would identify no ambiguous cases. Now work back from that extreme: gradually allow more A members to have FALSE values or more B members to be TRUE. You will gradually pick up more and more ambiguous cases. This leads to an obvious question: what is the tradeoff between "closeness" to the other Group and the number of ambiguous cases that you want to identify? If you only want to identify a very small number then emphasis on "big" differences might be tolerable (perfect univariate separation is unlikely), but otherwise an emphasis on "big" inter-Group differences might get in the way. There also is the issue if you care more about identifying ambiguous cases in one Group versus the other, which might need to enter into the analysis.

Second, for "univariate": looking at these variables one at a time limits the possibility of identifying ambiguous cases based on combinations of variable values. Your interest in getting individual cases for examining correlations indicates that you already appreciate this limitation.

Logistic regression

Although you say "I'm not trying to predict group membership, just describe it," developing a formula to predict group membership might be a better approach to identifying ambiguous cases. A logistic regression model based on individual data would determine the relation between a linear combination of predictor variables and the log odds of membership in Group B versus Group A.

This could be useful in either of 2 potential uses of this model. If incoming cases don't yet have known group membership but have known values of the predictor variables, the most ambiguous cases would be those for which the predicted odds ratio of membership is close to 1. If instead incoming cases are already labeled by Group and you want to find those in Group A who are closest to Group B, then Group A cases having values of the linear predictor most strongly indicating membership in Group B are the ones you want to identify.

Note that a logistic regression would also answer the question as originally posed. Each predictor variable in the logistic regression model, whether continuous or categorical, would have an associated regression coefficient and standard error expressed in comparable terms. The largest individual Wald statistics, the square of the ratio of each coefficient to its standard error, would indicate the variables with the "biggest" relations to Group membership in one well-defined way, as considered in the related context of survival analysis on this Cross Validated page. And the logistic regression would have the advantage of determining this relative importance for each variable with all the other variables taken into account.

Implementation issues

For this application of logistic regression, with a large number of predictor variables, you will have to take some care to avoid perfect separation based on combinations of variables, particularly since you are interested in the ambiguous cases rather than in perfecting prediction of Group membership. Some ways to deal with this problem are discussed on this Cross Validated page and its links, and books like Regression Modeling Strategies provide approaches for reducing the set of predictors.

Also you will need to validate your model in some way. You may have enough cases to set aside separate training, test, and validation sets, but at the minimum you will have to perform cross validation or bootstrapping to establish that you have a generally applicable model rather than one that just happens to match your particular data sample (however large it may seem). It can be amazing to see how different the sets of "biggest" variables can be when based on different bootstrap samples from the same data set. Validation needs to be addressed regardless of whether you use the logistic regression approach or some other scheme.

Univariate approaches

If you are nevertheless forced to use the aggregated rather than individual data, the approaches you outline in the question are heading in the right direction. Note that these approaches implicitly use the Group membership as the predictor variable in a large set of multiple univariate comparisons.

For continuous variables, you proposed a Z-test statistic (ratio of difference between group means to the standard error of that difference) as a measure of how "big" the effect is. This defines "big" in terms of your ability to detect a difference between the two Groups (given the variability and the numbers of cases), which makes sense.

For 2 x 2 categorical comparisons, the log odds ratio and its standard error (as might come from logistic regression) can similarly be used to form a Z-test statistic, as they also take a normal distribution in the limit. These are easily calculated from a contingency table. If any of your putative predictor variables are categorical with 3 or more levels, the multinomial regression of that variable against Group can be set up as a set of independent binary comparisons to get similar statistics.