Comparing ways to create a composite score

Question

Objective:

I have biomarkers $X_1,\ldots,X_p$ (all in continuous scale) and a binary dependent variable $Y$. Because $p$ is large (there are many biomarkers), I want to make a composite score combining $X_1,\ldots,X_p$. However, not all the biomarkers are expected to be related to $Y$ and I don't want to include the unrelated biomarkers to create my composite variable. I'll use this composite variable in a regression of $Y$ with other covariates to see if these selected biomarkers jointly show any association to $Y$.

Problems:

1) The scale and variance of the biomarkers differ a lot.

2) All biomarkers have skewed distributions.

3) I have decided to include those biomarkers to create the composite variable for which the bivariate associations to $Y$ are significant ($p<0.05$). But sometimes the Wilcoxon test shows a biomarker is not significant ($p>0.05$) but the univariate logistic regression (when only one biomarker is used as the predictor) shows it is significant ($p<0.05$), and vice versa. Sometimes the p-values were drastically different.

Question 1: Which p-value should I use (Wilcoxon test vs. univariate logistic regression) to decide which biomarkers to include in the composite creation (and why)?

Methods:

1) After we can decide which biomarkers to include in the composite, we can see the direction of the association (in our case higher biomarker values are related to $Y=1$ for all biomarkers), find quartiles, and sum together the quartile ranks to create a simple composite variable.

2) We can extract the first principal component score and use that as a composite variable.

3) We can extract the $\beta$ coefficients from the univariate logistic regressions for each of the (standardized) biomarkers, then multiply those with the (standardized) biomarker levels to create a composite.

4) Extract the $\beta$ coefficients from the multivariable logistic regressions with all (standardized) biomarkers and then multiply those with the (standardized) biomarker levels to create a composite.

Question 2: Do you see any problem with the 3rd or 4th method?

Validation:

We are planning to compare these different methods of composite variable creation by regressing the composite variables separately (along with other covariates) and finding out the AUC of the models. The best method to create the composite will the one that produces the highest AUC.

Question 3: Is this method valid for comparison? Is there an issue with comparability of these three methods? Is there a better method that we can consider?

Answer 1

Question 1: You are incorrect that "we don't need any distributional assumption for the biomarkers in logistic regressions." A single-predictor logistic regression specifically assumes that the log-odds of the binary outcome are linearly related to the values of the predictor. So if you are using, say, RNAseq data as predictors you will get different results (for coefficients and p-values) if you use sequence counts instead of log-transformed counts.

It is not at all surprising that logistic regression, with that strong parametric assumption, and the non-parametric rank-based Wilcoxon test are giving different p-values. Logistic regression might be more powerful (better ability to detect true significant associations) when the linearity assumption is met, but not when the assumption is violated. The validity of the linearity assumption might differ among predictors.

That said, you should be wary of using any set of single-predictor tests to select components for your composite score. Logistic regression has an inherent omitted-variable bias such that if you omit any predictor related to outcome from a model you will bias the coefficients of the includes predictors. See this answer and its links as one of many on this site that discuss these dangers.

Question 2: Based on the above, your Method 3 has substantial problems as it relies on a whole set of logistic regressions each of which omits many predictors related to outcome. The fourth method would be preferable, but a related approach described below could be even better, depending on the scale of your problem.

Question 3: Although AUC is better than some measures of model performance, it has significant drawbacks for model comparison. The best way to evaluate a model that predicts a probability of an outcome is to use a proper scoring rule like the Brier score. You also need to be thorough in how you perform your comparisons. You should be evaluating each entire model-building process starting from the initial data, with bootstrapping or cross-validation, particularly when your modeling used the outcomes to select the predictors.

Alternate approaches: These depend on whether you are evaluating a few dozen potential predictors (as in some clinical studies) or thousands of them (as in RNAseq studies).

In the first case you should consider approaches like those recommended by Harrell's Regression Modeling Strategies. Chapter 11 of the second edition is a clinical case study that illustrates how to perform data reduction (including linear and nonlinear principal components), selection among modeling variations, backward variable selection from a full model to simplify, and model evaluation for logistic regression.

In the second case you should be using a principled way to select and weight the predictors for a composite score. LASSO comes immediately to mind. This can be thought of as starting with the best individual predictor but then adding additional predictors in a way that avoids the overfitting seen in standard stepwise approaches. An Introduction to Statistical Learning provides one accessible presentation in Chapter 6 with a worked example for standard linear regression, but the glmnet() function illustrated there (of the R package having the same name) also allows for logistic regression. This would provide you with something similar to your Method 4 in Question 2, but with a more reliable basis. You might also consider the Elastic Net, a combination of LASSO and ridge regression, that minimizes the instability in LASSO predictor selection when there are multiple correlated predictors. Statistical Learning with Sparsity describes Elastic Net starting in Chapter 4. Elastic Net can also be implemented via glmnet().

A final warning: if you are going to use any of these linear regression approaches you need to document the linear relationship between the predictors and the log-odds of outcome. I suspect that a failure of that linear relationship for some of your candidate predictors led to your original question about different results with logistic regression and Wilcoxon test results, so you don't want to face that problem again farther down the road.