Approaches to compare differences in means with differences in proportions?

Question

I have summaries of hundreds of variables for two populations. Most of these variables are counts, but some are continuous. For example, if I had population sizes $N_0 = 20000$ people and $N_1 = 10000$, I might have a:

• Count variable: people with red hair. There are $X_0 = 2000$ and $X_1 = 2500$ redheads in populations 0 and 1, respectively.

• Continuous variable: age. In population 0, average age is $\bar{X_0} = 52$ with standard deviation $S_0 = 17$. In population 1, $\bar{X_1} = 62$ and $S_1 = 16$

And my file might look like:

library(data.table)

(d <- data.table(var_type = c('bin', 'cont'), var_name = c('redhead', 'age'),
                 n_0      = c(2e4L,    2e4L), n_1      = c(1e4L,       1e4L),
                 x_0      = c(2e3L,     NA), x_1       = c(2500L,        NA),
                 avg_0    = c(NA,       52), avg_1     = c(NA,           62),
                 sdev_0   = c(NA,       17), sdev_1    = c(NA,           16)
                 ) )
#    var_type var_name   n_0   n_1  x_0  x_1 avg_0 avg_1 sdev_0 sdev_1
# 1:      bin  redhead 20000 10000 2000 2500    NA    NA     NA     NA
# 2:     cont      age 20000 10000   NA   NA    52    62     17     16

I am supposed to answer the question What are the variables with the biggest differences between these populations? where biggest is not at all well-defined.

I have pointed out that this definition and the criteria for comparison between the count and continuous variables are not clear, and got the response, Come up with something. Thus I've attempted the following:

Standardized Mean Difference: For the count variables, I create $\bar{X}$ and $S$ by treating each observation as a Bernoulli outcome.

# p = x / n
d[var_type=='bin', `:=`(avg_0 = x_0/n_0, avg_1 = x_1/n_1)] 
# s = sqrt(p*(1-p))
d[var_type=='bin', `:=`(sdev_0 = sqrt( avg_0 * (1-avg_0) ),  
                        sdev_1 = sqrt( avg_1 * (1-avg_1) ) )]    
# assuming equal population variances
d[, sdev_pool := sqrt( (  (n_0 - 1) * sdev_0 ^ 2 + (n_1 - 1) * sdev_1  ^ 2 ) /
                          (n_0 + n_1 - 2) ) ]    
d[, smd := (avg_1 - avg_0) / sdev_pool][]    
#    var_type var_name ... avg_0 avg_1 sdev_0     sdev_1  sdev_pool       smd
# 1:      bin  redhead ...   0.1  0.25    0.3  0.4330127  0.3499985 0.4285733
# 2:     cont      age ...  52.0 62.00   17.0 16.0000000 16.6733430 0.5997597

NHST: At the risk of being ostracized from CV... yes, I did this. Since the $\chi^2$ test with df=1 is equivalent to the $z$-test, I can use the same test for both variable types. The results are not blindingly illuminating.

# use pooled std err
d[, z := (avg_1 - avg_0) / (sdev_pool * sqrt(1/n_0 + 1/n_1)  )]
# 2-tailed test
d[, p_val := 2*pnorm(-abs(z))][]
#    var_type var_name   n_0   n_1  ... avg_0 avg_1 ...  sdev_pool ...        z         p_val
# 1:      bin  redhead 20000 10000  ...   0.1  0.25 ...  0.3499985 ... 34.99287 2.888512e-268
# 2:     cont      age 20000 10000  ...  52.0 62.00 ... 16.6733430 ... 48.97018  0.000000e+00

Using this approach, one concludes that the difference in age is slightly more significant (clinically or statistically) than the difference in redheads. Although I feel like the latter is a "bigger" effect size, I can't quantify it because I can't make an equivalent of an odds ratio (or arcsine transform, or logit, etc...) for the continuous outcome.

Discretization: I haven't tried this yet, but I guess I could discretize the continuous variables and compare counts in each "bin" between groups. In other words, compare the counts of 55- to 65-year-olds, etc. That is probably what I will wind up doing in the absence of better ideas.

My question (finally!) is for insights to help define bigness of difference and approaches to systematically compare differences across these variable types.

Answer 1

Goal of project

According to your comment the ultimate goal is "to create a scoring formula to identify members of one population who most closely resemble the other population" (emphasis added). Let's call those the "ambiguous" cases, and call the populations "Group A" and "Group B."

Problems with proposed approach

Looking for the biggest univariate differences (however defined) between the Groups might not be what is needed to identify ambiguous cases.

First for "biggest": consider the extreme case of perfect separation, a logical variable that is TRUE for all members of Group A, but FALSE for all members of Group B. That would score best of all as a "biggest" difference, but would identify no ambiguous cases. Now work back from that extreme: gradually allow more A members to have FALSE values or more B members to be TRUE. You will gradually pick up more and more ambiguous cases. This leads to an obvious question: what is the tradeoff between "closeness" to the other Group and the number of ambiguous cases that you want to identify? If you only want to identify a very small number then emphasis on "big" differences might be tolerable (perfect univariate separation is unlikely), but otherwise an emphasis on "big" inter-Group differences might get in the way. There also is the issue if you care more about identifying ambiguous cases in one Group versus the other, which might need to enter into the analysis.

Second, for "univariate": looking at these variables one at a time limits the possibility of identifying ambiguous cases based on combinations of variable values. Your interest in getting individual cases for examining correlations indicates that you already appreciate this limitation.

Logistic regression

Although you say "I'm not trying to predict group membership, just describe it," developing a formula to predict group membership might be a better approach to identifying ambiguous cases. A logistic regression model based on individual data would determine the relation between a linear combination of predictor variables and the log odds of membership in Group B versus Group A.

This could be useful in either of 2 potential uses of this model. If incoming cases don't yet have known group membership but have known values of the predictor variables, the most ambiguous cases would be those for which the predicted odds ratio of membership is close to 1. If instead incoming cases are already labeled by Group and you want to find those in Group A who are closest to Group B, then Group A cases having values of the linear predictor most strongly indicating membership in Group B are the ones you want to identify.

Note that a logistic regression would also answer the question as originally posed. Each predictor variable in the logistic regression model, whether continuous or categorical, would have an associated regression coefficient and standard error expressed in comparable terms. The largest individual Wald statistics, the square of the ratio of each coefficient to its standard error, would indicate the variables with the "biggest" relations to Group membership in one well-defined way, as considered in the related context of survival analysis on this Cross Validated page. And the logistic regression would have the advantage of determining this relative importance for each variable with all the other variables taken into account.

Implementation issues

For this application of logistic regression, with a large number of predictor variables, you will have to take some care to avoid perfect separation based on combinations of variables, particularly since you are interested in the ambiguous cases rather than in perfecting prediction of Group membership. Some ways to deal with this problem are discussed on this Cross Validated page and its links, and books like Regression Modeling Strategies provide approaches for reducing the set of predictors.

Also you will need to validate your model in some way. You may have enough cases to set aside separate training, test, and validation sets, but at the minimum you will have to perform cross validation or bootstrapping to establish that you have a generally applicable model rather than one that just happens to match your particular data sample (however large it may seem). It can be amazing to see how different the sets of "biggest" variables can be when based on different bootstrap samples from the same data set. Validation needs to be addressed regardless of whether you use the logistic regression approach or some other scheme.

Univariate approaches

If you are nevertheless forced to use the aggregated rather than individual data, the approaches you outline in the question are heading in the right direction. Note that these approaches implicitly use the Group membership as the predictor variable in a large set of multiple univariate comparisons.

For continuous variables, you proposed a Z-test statistic (ratio of difference between group means to the standard error of that difference) as a measure of how "big" the effect is. This defines "big" in terms of the magnitude of difference between the two Groups relative to the error in the estimate of the difference. That's a perfectly reasonable definition.

For 2 x 2 categorical comparisons, the log odds ratio and its standard error (as might come from logistic regression) can similarly be used to form a Z-test statistic, as they also take a normal distribution in the limit. These are easily calculated from a contingency table. If $A,B$ represent the Groups; 1,2 represent the two classes of a categorical variable; and $N$ represents the number of cases with the indicated combination of Group and class, then the log odds ratio is:

$$\log\left(\frac{N_{A1}N_{B2}}{N_{A2}N_{B1}}\right)$$

with approximate standard error:

$$\sqrt{\dfrac{1}{N_{A1}} + \dfrac{1}{N_{A2}} + \dfrac{1}{N_{B1}} + \dfrac{1}{N_{B2}}}$$

If any of your putative predictor variables are categorical with 3 or more levels, the multinomial regression of that variable against Group can be set up as a set of independent binary comparisons to get similar statistics.

That way you have comparable measures of "big" for continuous and categorical variables.