Is the use of cutoffs for dichotomisation of biomarkers really that bad?

Question

Tissue microarrays are commonly used to assess potential prognostic biomarkers. For decades now, many authors (I would even say the majority) feel the need to categorise their continuous predictors, either by dichotomising after data collection, or by using scoring systems that pre-categorise (e.g. Allred and immunoreactive score).

The pitfalls of this approach are numerous and have been highlighted for years. Douglas Altman and Patrick Royston have been some of the most vocal critics, alongside prediction model experts like Frank Harrell.

I completely agree with their arguments, but does it really matter in practice? A frequent defence of dichotomisation, or categorisation of continuous predictors in general, is that it greatly simplifies clinical decision making. Indeed, a physician may have to weigh up what is essentially a binary choice – to treat or not to treat; or to present the patient with one prognosis or another – and so the belief is that predictors should be dichotomised to reflect this. I would argue that keeping a predictor continuous, however, allows for more granular information to be obtained and lessens the risk of misclassifying patients whose predictor values lie just above or below the arbitrary threshold.

However, countless clinical decisions are made off back of categorised predictors. For example, breast cancer patients are assessed for oestrogen, progesterone and HER2 expression status and are classified as receptor positive or negative as per some threshold. Although this approach may not be 'statistically sound', its widespread use in guiding decisions over treatment surely points to categorisation at least being somewhat useful?

Answer 1

It's really that bad, as detailed in many of Patrick and my writings. Think about it these ways, for starters: If you categorize a marker the loss of information is so great that you will need to go an collect more markers to make up for the loss. Why not instead get the most information out of a single marker? Then there is the issue that it is easy to show that the threshold for a marker is a function of the continuous values of all the other predictors. See the Information Loss chapter in BBR. For example, if the sex of the patient is an important factor with regard to the likelihood of disease, you'll find that a different biomarker threshold is needed for females vs. males.

Categorization simplifies decision making but only by making it worse. Many things only seem to be simple precisely because they are wrong.

We are taught to assess goodness of fit of models. A model that assumes a piecewise flat relationship between a biomarker and outcome will be easily shown to have poor fit to the data. For example look at the log-likelihood for a piecewise flat model (i.e., one using thresholding) and a flexible but everywhere-smooth model.

Answer 2

Although cutoffs in biomarker values are admittedly not helpful for modeling within a single population (+1 to Frank Harrell's answer on that basis), they make sense when, together with extensive subject-matter knowledge, they can delineate fundamentally different populations that benefit from different therapy. In such situations, testing can classify most cases with high probability (based on separate high and low cutoffs) while assigning intermediate cases to further testing or more cautious application of the findings. Breast cancer, the example of the OP in which tumors are evaluated for estrogen receptor alpha (ER), progesterone receptor (PR) and human epidermal growth factor receptor 2 (HER2), is such a situation.

This hexbin plot illustrates the 2-dimensional distribution of the expression of HER2 versus ER among 1093 primary breast cancers, based on RNA sequencing data from The Cancer Genome Atlas. Note the logarithmic scales on both axes.

Density plots for ER and HER2 make the class distinctions clearer.

ER shows a clearly bimodal distribution with modal values differing by 2 orders of magnitude and few tumors having intermediate values. HER2 values show a major peak encompassing about 90% of tumors, but about 10% show substantially larger expression, up to 20-fold more than that of the major peak.

Decades of clinical and laboratory research have shown that these distributions represent fundamentally different biological classes of breast cancer, driven by different tumorigenic processes and responding best to different types of therapy. For example, with respect to tumors having high expression of HER2 (also called ERBB2), Wikipedia summarizes:

Amplification, also known as the over-expression of the ERBB2 gene, occurs in approximately 15-30% of breast cancers. It is strongly associated with increased disease recurrence and a poor prognosis; however, drug agents targeting HER2 in breast cancer have significantly positively altered the otherwise poor-prognosis natural history of HER2-positive breast cancer... HER2 is the target of the monoclonal antibody trastuzumab (marketed as Herceptin). Trastuzumab is effective only in cancers where HER2 is over-expressed.

Similarly, patients with high ER expression can benefit from "endocrine therapy" that interrupts cellular signaling pathways mediated by estrogens. But cases at the lower left of the hexbin plot represent "triple-negative" breast cancers, low in all of ER, PR and HER2, that do not respond to therapies that target either estrogen signaling or HER2 signaling.* They need different therapies.

In practice, expression of ER, PR and HER2 is evaluated at the protein level by immunohistochemistry (IHC) rather than by the RNA sequencing that provided data for this plot. IHC makes at best semi-quantitative assessments of protein expression. For example, the Allred score cited by the OP combines the intensity and extent of IHC staining into an 8-point scale.

Nevertheless, given the large differences between high- and low-expressing tumors with respect to levels of these particular proteins, it's generally possible to assign a tumor to one of those biologically distinct classes based on IHC. The current recommendation of the College of American Pathologists (CAP) is to use the percentage of tumor-cell nuclei showing ER expression as the criterion, with tumors having <1% called ER-negative and those with >10% ER-positive. Tumors having 1% to 10% on that scale are reported as "ER Low Positive" with specific cautions about therapeutic choices that might be based on that finding. Similar guidelines for HER2 suggest an additional type of assay, in-situ hybridization, when IHC results are equivocal.

In that respect, behavior in practice follows what Harrell says on page 258 of Regression Modeling Strategies: "Physicians sometimes profess to desiring a binary decision model, but if given a probability they will rightfully apply different thresholds for treating different patients or for ordering other diagnostic tests." Based on clinical outcome studies, the CAP guidelines identify levels of expression that have high probability of correct classification, and recommend further tests and caution otherwise.

Finally, the classifications based on ER, PR, and HER2 expression are only part of what's considered in choosing therapy. Those are considered together with the tumor's histological appearance and size, whether it has spread to lymph nodes or beyond, further gene-expression or genetic tests, and patient preferences for breast conservation to select a combination of surgery, radiation, and pre-operative and adjuvant (post-surgical) drug therapies. The National Comprehensive Cancer Network publishes a 200+ page book of Guidelines for breast cancer therapy based on such considerations, documenting the level of evidence behind its recommendations, with 700+ references to the literature.

Data and code:

From Firebrowse, select BRCA for breast cancer, click on the mRNASeq bar, and select illuminahiseq_rnaseqv2-RSEM_genes_normalized. Restrict to primary tumors (sample type 01). Put into a data frame, here called brcaRNAseqPrimDF. Then in R:

library(ggplot2)
## for hexbin
ggplot(brcaRNAseqPrimDF, aes(x=ESR1.2099, y=ERBB2.2064)) + scale_x_log10() + scale_y_log10() + geom_hex(bins=50) + xlab("ER") + ylab("HER2") + theme(aspect.ratio = 1)
## for density plots
pHER2 <- ggplot(brcaRNAseqPrimDF,aes(x=ERBB2.2064))+scale_x_log10()+geom_density()+xlab("HER2")+geom_rug(alpha=0.2)
pESR1 <- ggplot(brcaRNAseqPrimDF,aes(x=ESR1.2099))+scale_x_log10()+geom_density()+xlab("ER")+geom_rug(alpha=0.2)
library(gridExtra)
grid.arrange(pESR1,pHER2,ncol=2)

ESR1 is Estrogen Receptor alpha. ERBB2 is the currently accepted name for HER2.

---

*For simplicity I'm not trying to incorporate the third classic immunohistochemical marker, prostaglandin receptor, as essentially all cases with low ER also have low prostaglandin receptor expression (called PGR.5241 in the data).