For a disease that I am interested in, five or six published works of research have shown an association of changes in gene expression with presence of the disease. However, all these diagnostic biomarker studies were performed with small numbers of cases (15-30) and controls (15-30), and the overlaps between their lists of potential biomarkers are minimal. The cases and controls are defined using clinically relevant criteria.

We recently completed a relatively large, multi-institutional study with 80 cases and 80 controls to look into this. All clinical specimens were processed together as a batch at one of the institutions. Analyses of our data suggests that there is no association of gene expression with presence of the disease. Using data from one of the published studies that used the same gene expression measurement technology as this study, to calculate effect-size with the SSPA R package based on moderated t statistics as per the limma R package, this study has a power of ~92% at alpha=0.10.

Analyses that we have performed are:

  1. unsupervised hierarchical clustering based on various kinds of distance measurements (subjective evaluation)

  2. principal component analysis; traditional and matrix plots (subjective evaluation)

  3. various kinds of two-group tests for differential expression of genes (significance threshold of 0.05 for P values adjusted for false discovery)

  4. classification/prediction analyses with leave-one-out and Monte Carlo cross-validations using linear kernel support vector machines (SVM) and top-scoring pair (TSP) methods; for SVM, variable selection is based on differential expression measures; TSP does not require differential expression (accuracy rates in these analyses are in the 45%-55% range)

  5. confirming absence of differential expression for some of the genes, identified in the previous studies as putative disease markers, after quantifying gene expression using a different technology (reverse transcription-PCR instead of hybridization microarray)

Are these enough to have confidence in the negative finding of our study, that gene expression measurements cannot be used for disease diagnosis? Is there any other type of subjective or objective evaluation of the expression data-set that I should do?

I understand the usual caveats for such conclusions. E.g., there might be sub-groups of cases where disease markers may exist, or that the sensitivity or specificity of the measurement technology may not have been adequate enough for some true marker genes to be picked up.

Thank you.


"Are these enough to have confidence in the negative finding of our study" - it depends on what you mean by "have confidence". Can you walk away and say "The association is negative, we're done here". No. You can be confident that, having looked at it several ways, you're not detecting an association in your data.

But confidence in the negative finding as a reflection of "truth"? Not really. The confidence one should have in your findings aren't a function of the sheer amount of analysis you throw at it. One could, for example, use ever more elaborate regression techniques to look at their data set in entirely new and novel ways, but if their subjects are misclassified, or they've missed a major confounding variable, then they simply have an impressive volume of incorrect results.

There are other considerations as well. Your study may be underpowered. Smaller studies are more underpowered, but it's possible to be both underpowered and lucky. Similarly, you may simply be experiencing some amount of random variation - some number of studies looking at a "Capital-T Truth, God's Eye View" positive effect will still find negative or null findings. It's possible you study was one of these.

If we assume that your data was collected correctly however, and bias from things like misclassification and confounding are minimal, I think you can be confident in your finding for your study population, and you can be confident that your findings suggest that the association is negative. But a sheer mass of analysis on a single study population does not a body of evidence make.

  • $\begingroup$ Thank you for the comments. I have added to the question that the this is a multi-institutional study with clinically rational criteria for defining cases and controls. I have also provided results of power analysis. I understand the larger point you make, but reviewers of the study will certainly like to know the results of some types of analyses. E.g., a differential expression analysis is definitely expected. $\endgroup$ – user4045 Sep 21 '12 at 6:54
  • $\begingroup$ +1 for some good comments but I think more can be said and some things I havein mind are positive and constructive. $\endgroup$ – Michael Chernick Sep 21 '12 at 10:27

This is a really great question! Negative studies need to be published more often in the literature to reduce or eliminate publication bias. I like it that you have done so many thoughtful analyses.

EpiGrad provides some good caveats in his answer. However I think there is a glass half full way to look at this that should be considered. One thing that I did not see on your list was a meta analysis. I think you could do one and should add all the well designed and comparable studies (having negative or positive results). If the analysis leads to a negative conclusion I think it strengthens your case.

Also tradition has it that in classical hypothesis testing the negative result is the null hypothesis and the alternative is the positive result that the investigator is trying to provide statistical inference to reject this null hypothesis. Studies are designed to have sample sizes that maximize the power (making it likely that you will reject the null hypothesis when your alternative is correct).

In pharmaceutical clinical trials the FDA will sometimes give approval to drugs that are shown to be non-inferior to a control (or equivalent in other cases such as approval of a generic drug). These studies involve reversing the null and alternative hypotheses. These studies are powered to reject inferiority (or non-equivalence). If you can look at the data this way and reject a null hypothesis that the biomarker works I think that will overcome EpiGrads valid criticism.

Because the Neyman-Pearson approach to hypothesis tesing is controlled by sample size considerations to reject the null hypothesis through the power of the test and is not designed to "prove" the null hypothesis statistician always warn users that if the null hypothesis is not reject you should not conclude that the null is accepted. Switching the null and alternative finesses this problem.

  • $\begingroup$ I thank you for the valuable insight about how the hypothesis of a study is framed. For the scenario I have in the question, there is no known biological/mechanistic basis to connect gene expression patterns with presence of the disease. The observations of the other studies suggest this association (because of which the larger study was done), but they do not provide a mechanistic basis. There are possibilities, based on experimental facts, that one can link together to guess a mechanistic basis for the connection, but it is also equally easy to cast doubt on such a connection. $\endgroup$ – user4045 Sep 21 '12 at 11:23
  • $\begingroup$ @user4045 Do you see no way to do a reasonable meta-analysis? $\endgroup$ – Michael Chernick Sep 21 '12 at 11:46
  • 1
    $\begingroup$ @user4045 Although not used much in situations like these Fisher's combination test can be used. If you are not familiar with it what it does is to take the individual p-values for the different studies (so you don't even have to be applying the same test or have very similar designs) and computes a combined pvalue. I think really what is important is that they are basically testing the same hypothesis in a general sense in your case whether or not there is an association between gene expression for your particular gene and the presense of the disease in the subject. $\endgroup$ – Michael Chernick Sep 21 '12 at 14:42
  • 1
    $\begingroup$ @whuber One very important key to doing a meta analysis is to select the appropriate set of studies. The current study contradicts others. The OP may know why in some cases and may be able to put together a rational argument for which studies to include and which studies not to. If the studies are chosen properly a meta-analysis will improve things because it takes into account the synthesis of good data from those several studies. We should not presume that the conclusions of the new study are correct. If the results hold up after a well designed meta analysis the evidence is stronger. $\endgroup$ – Michael Chernick Sep 21 '12 at 15:51
  • 1
    $\begingroup$ @whuber I should have added in my remarks that if there are no studies that could be considered reputable and satisfy a rational criteria for inclusion then no meta-analysis can be done at least until future studies come along. I think given that staement we probably are really in agreement. $\endgroup$ – Michael Chernick Sep 21 '12 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.