The relationship between prediction performance and the degree of association between variables?

Question

I've been reading a paper (Machine learning identifies candidates for drug repurposing in Alzheimer’s disease), but having a hard time understanding its core idea. Basically, the researchers

1. Obtained a dataset of postmortem brain tissues that includes the expression values of ~20k genes as features and disease stage (early, intermediate, late) as the target.
2. Applied 80 drugs to cell cultures and recorded the differentially expressed (either upregulated or downregulated vs control) genes for each drug.
3. For each drug, they limited the feature space (all ~20k genes from the brain tissue data) to only the drug associated gene list; then trained and validated "predictors" or "classifiers".
4. Tried different algorithms for the ML work (logistic regression, SVM, boosted random forest, neural network) and decided on logistic regression based on its AUC.
5. Produced empirical p-values for each drug by comparing the AUC of the predictor of the drug and 1000 size-matched gene lists. For example, if ruxolitinib has an empirical p-value of 0.004, and caused the perturbation of 300 genes, then that means the logistic regression model fit by genes expressed after ruxolitinib application predicts the disease stage better than (has a higher AUC than) 996 randomly selected lists of 300 genes.
6. Ranked drugs based on their empirical p-values and claimed that "If a classifier trained on the expression of genes associated with a particular drug is substantially more accurate than equivalent classifiers trained on the expression of any arbitrarily chosen genes, then such a result suggests that the drug-associated genes carry at least some disease-related signal."

Now what I don't understand is, how can they go from this line of reasoning to claiming that the pharmacological mechanism of action of high ranking drugs have a higher possibility of overlapping with the pathological mechanisms of Alzheimer's disease? How can merely the list of genes perturbed by a certain drug predicting the disease stage better compared to random lists provide evidence for the drug being associated with the disease? From the article

We present DRIAD (Drug Repurposing In AD), a machine learning framework that quantifies potential associations between the pathology of AD severity (the Braak stage) and molecular mechanisms as encoded in lists of gene names.

Could it actually be the approach some statistics blogs & textbooks recommend 2 3 when they talk about using predictive statistics (predictive power score, to be precise) instead of merely testing for correlation?

Answer 1

There's both a statistical and a biological aspect to this question.

The statistical aspect is reasonably straightforward. You want to find some way to evaluate whether an association of a set of predictors with outcome might just be due to chance. "Significance" is evaluated as whether results are adequately different from what might be expected by chance under a "null hypothesis" of no true association.

Start with a single gene set. It doesn't matter how it was identified. If a set of 300 genes seems to be associated with outcome in a statistical model, how do you compare that association against a "null hypothesis" that the association with outcome is just due to chance?

There are several ways to do this.* Some ways have a theoretical basis. For example, you can evaluate the "significance" of a logistic-regression model by tests based on the underlying theory of maximum-likelihood estimation. The problem is that when you have a large number of predictors relative to the number of cases you can overfit the current data in a way that will appear "significant" but won't generalize well to new data.

One way the authors dealt with that was to use a type of "penalized" logistic regression, ridge regression, to minimize the risk of overfitting. That, however, removes the simple theoretical basis for estimating "significance."

The authors chose a different way to evaluate what you might find under the null hypothesis of no association of the gene set with outcome. Any random set of 300 genes (out of ~20,000 total) is unlikely to be truly associated with outcome, but there will be variability among such random sets of genes in their apparent associations with outcome. Multiple sets of 300 randomly chosen genes (these authors used 1000 sets) thus provide the variability of associations you might find under the null hypothesis.

The "significance" test is then how unusual it is for a randomly chosen set of 300 genes to be as strongly associated with outcome as the set that you've identified. In your example, only 4 out of 1000 random sets of 300 genes were that strongly associated with outcome, for $p=0.004$. Thus there's a very low probability that the association of the identified set of genes with outcome was simply due to chance.

Ranking drugs by the p-values of their altered gene sets is perhaps a bit more problematic, as I suspect that different numbers of genes in the gene sets (and thus in the sizes of the corresponding random gene sets) might lead to extra variability in the p-value estimates. It seems like a reasonable heuristic, however. You might have more confidence in a drug whose gene set that has only 4/1000 chance of a random association with outcome than one that has a 40/1000 chance.

There might be some quibbles about details, but the fundamental approach is reasonable.

The biological aspect is that having sets of genes or rankings of drugs, as DRIAD provides, is only a first step in investigation. All you can say from DRIAD itself is that sets of genes whose expression levels are affected by some drugs in vitro are also associated with the severity of Alzheimer's, and that some drugs affect expression of genes that are more strongly associated with severity than others.

In fairness, the authors don't claim that any drug is itself associated with the disease; the quote that you include near the end of the question is pretty cautious.

In the rest of the report, the authors used the DRIAD drug ranks as the basis of further pharmacologic investigation. The drugs were all designed to be inhibitors of kinases (enzymes that phosphorylate molecules, typically proteins). Inhibiting kinases is presumably what led to the drugs affecting the expression of their gene sets. Kinase inhibitors, however, typically affect multiple related kinases. So the authors went on to identify the affinities of the top-ranked drugs for multiple kinases (Figure 4) and to examine which specific kinases tended to be inhibited by multiple high-ranking drugs (Figure 5). That led to identifying 10 top kinases that might be most closely associated with the biological pathways that define the difference between early- and late-stage Alzheimer's (Table 1).

You are correct to be skeptical of any report that is solely based on gene sets or drug rankings. In this case, however, the gene sets and associated drug rankings by DRIAD just provided a start for further detailed pharmacologic analysis.

---

*This has nothing to do with the "predictive power score" that you cite at the end of the question. That's a way to use a very flexible fit to see if there is any association between two variables, an association that might be highly nonlinear and thus missed by a linear correlation or linear regression. The logistic regression used by the authors of this study is just a fit of the log-odds of the binary severity outcome against a strictly linear combination of (log-transformed) gene-expression values.