Claiming validity of a study's negative finding

Question

For a disease that I am interested in, five or six published works of research have shown an association of changes in gene expression with presence of the disease. However, all these diagnostic biomarker studies were performed with small numbers of cases (15-30) and controls (15-30), and the overlaps between their lists of potential biomarkers are minimal. The cases and controls are defined using clinically relevant criteria.

We recently completed a relatively large, multi-institutional study with 80 cases and 80 controls to look into this. All clinical specimens were processed together as a batch at one of the institutions. Analyses of our data suggests that there is no association of gene expression with presence of the disease. Using data from one of the published studies that used the same gene expression measurement technology as this study, to calculate effect-size with the SSPA R package based on moderated t statistics as per the limma R package, this study has a power of ~92% at alpha=0.10.

Analyses that we have performed are:

1. unsupervised hierarchical clustering based on various kinds of distance measurements (subjective evaluation)

2. principal component analysis; traditional and matrix plots (subjective evaluation)

3. various kinds of two-group tests for differential expression of genes (significance threshold of 0.05 for P values adjusted for false discovery)

4. classification/prediction analyses with leave-one-out and Monte Carlo cross-validations using linear kernel support vector machines (SVM) and top-scoring pair (TSP) methods; for SVM, variable selection is based on differential expression measures; TSP does not require differential expression (accuracy rates in these analyses are in the 45%-55% range)

5. confirming absence of differential expression for some of the genes, identified in the previous studies as putative disease markers, after quantifying gene expression using a different technology (reverse transcription-PCR instead of hybridization microarray)

Are these enough to have confidence in the negative finding of our study, that gene expression measurements cannot be used for disease diagnosis? Is there any other type of subjective or objective evaluation of the expression data-set that I should do?

I understand the usual caveats for such conclusions. E.g., there might be sub-groups of cases where disease markers may exist, or that the sensitivity or specificity of the measurement technology may not have been adequate enough for some true marker genes to be picked up.

Thank you.

Answer 1

"Are these enough to have confidence in the negative finding of our study" - it depends on what you mean by "have confidence". Can you walk away and say "The association is negative, we're done here". No. You can be confident that, having looked at it several ways, you're not detecting an association in your data.

But confidence in the negative finding as a reflection of "truth"? Not really. The confidence one should have in your findings aren't a function of the sheer amount of analysis you throw at it. One could, for example, use ever more elaborate regression techniques to look at their data set in entirely new and novel ways, but if their subjects are misclassified, or they've missed a major confounding variable, then they simply have an impressive volume of incorrect results.

There are other considerations as well. Your study may be underpowered. Smaller studies are more underpowered, but it's possible to be both underpowered and lucky. Similarly, you may simply be experiencing some amount of random variation - some number of studies looking at a "Capital-T Truth, God's Eye View" positive effect will still find negative or null findings. It's possible you study was one of these.

If we assume that your data was collected correctly however, and bias from things like misclassification and confounding are minimal, I think you can be confident in your finding for your study population, and you can be confident that your findings suggest that the association is negative. But a sheer mass of analysis on a single study population does not a body of evidence make.

Answer 2

This is a really great question! Negative studies need to be published more often in the literature to reduce or eliminate publication bias. I like it that you have done so many thoughtful analyses.

EpiGrad provides some good caveats in his answer. However I think there is a glass half full way to look at this that should be considered. One thing that I did not see on your list was a meta analysis. I think you could do one and should add all the well designed and comparable studies (having negative or positive results). If the analysis leads to a negative conclusion I think it strengthens your case.

Also tradition has it that in classical hypothesis testing the negative result is the null hypothesis and the alternative is the positive result that the investigator is trying to provide statistical inference to reject this null hypothesis. Studies are designed to have sample sizes that maximize the power (making it likely that you will reject the null hypothesis when your alternative is correct).

In pharmaceutical clinical trials the FDA will sometimes give approval to drugs that are shown to be non-inferior to a control (or equivalent in other cases such as approval of a generic drug). These studies involve reversing the null and alternative hypotheses. These studies are powered to reject inferiority (or non-equivalence). If you can look at the data this way and reject a null hypothesis that the biomarker works I think that will overcome EpiGrads valid criticism.

Because the Neyman-Pearson approach to hypothesis tesing is controlled by sample size considerations to reject the null hypothesis through the power of the test and is not designed to "prove" the null hypothesis statistician always warn users that if the null hypothesis is not reject you should not conclude that the null is accepted. Switching the null and alternative finesses this problem.