1
$\begingroup$

I have an experimental design problem and I'm not sure which would the best way to proceed.

We have a micro-array experiment in which we compare gene expression profiles between 2 groups of patients. Each group has 12 independent biological replicates. Lets call these groups CT and EXP. The goal of this experiment is to identify differentially expressed genes between CT and EXP.

Two sets of hybridizations where done: first and second. First hybridization had big issue: CT samples were processed at one time-point and EXP couple of months later, creating a perfectly confounded batch effect.

Second hybridization did not have this problem - all of the samples were processed at the same time.

I've analyzed both sets separately and came up with 2 lists of differentially expressed genes. My question is, should I discard the results of the first hybridization completely and stick with the second one since it didn't have an obvious design issue? Or should I use both sets of hybridizations and select only genes which show up as differentially expressed in every case (there are 2900 of such genes)?

Appreciate your help!

$\endgroup$
  • $\begingroup$ Are a subset of the results going to be verified using PCR or some other method? Or are the microarray results meant to stand on their own? $\endgroup$ – Livid Feb 6 '14 at 0:05
  • $\begingroup$ I was thinking that if I only use genes that are differentially expressed in both sets of replicates the PCR confirmation will no longer be necessary, since I can be fairly confident that they are truly differentially expressed. $\endgroup$ – slava Feb 6 '14 at 0:43
  • $\begingroup$ However if I only use the results from the second set, than I would definitely use RT-PCR to confirm. $\endgroup$ – slava Feb 6 '14 at 0:44
0
$\begingroup$

In my opinon you should PCR "confirm" a subset you think is likely interesting (based on theory/guessing) either way. There is a history of problems with microarrays giving "false positives". It is not just batch effects, but also position on the chips, baseline differences, etc. Also keep in mind that different expression profiles can result in the same phenotype, so only looking at averages rather than also looking for within-group clusters of similar profiles may be suboptimal. There are a number of reasons that the observed differential expression may not be clinically important. You may have reviewers who think the same. If you do the PCR verification then your conclusions are stronger regardless, just remember if you "verify" enough different genes some are bound to come up as significant. Since the study sounds exploratory (you have no theory predicting certain differences beforehand), the results should be presented as such.

Also, if both arrays are from the same two sets of subjects then it may be interesting to see which genes were not substantially different in one but were so in the other. Is there reason to expect the expression profiles to be stationary/stable over time? Out of curiosity, did you see more apparent group differences in the confounded experiment?

Edit in response to comments:

Interesting that nearly half the "group differences" can be accounted for by confounding and technical factors. I have no idea if this is the normal amount of variability seen for technical replicates of microarray data (never ran such an experiment myself). If it is much larger than usual it really calls the first study into question and really makes me wonder why there is such a large batch effect. On the other hand, if that is the normal amount of variability it makes me wonder what else is going on that affects the results.

Also if you are using significance testing to determine which genes are differentially expressed, keep in mind this warning by Gelman and Stern:

Consider two independent studies with effect estimates and standard errors of 25 ± 10 and 10 ± 10. The first study is statistically significant at the 1% level, and the second is not at all statistically significant, being only one standard error away from 0. Thus, itwould be tempting to conclude that there is a large difference between the two studies. In fact, however, the difference is not even close to being statistically significant: the estimated difference is 15, with a standard error of sqrt(10^2 + 10^2) = 14.

Andrew Gelman and Hal Stern. The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant. The American Statistician, November 2006, Vol. 60, No. 4

You got me looking up how variable microarray results are supposed to be and I found a few papers that look of interest. Zakharkin et al reported relatively consistent results for technical replicates. Also it may be helpful to create scatter plots comparing the actual expression levels for each replicate as their figure 4:

We found that technical replicates within a biological replicate had higher and more consistent correlations with each other than with other biological replicates. Generally, our correlations were higher than those observed by Dobbin et al., 2005, for interlaboratory correlations between tumor samples [25] and were compatible with values for in-lab correlations obtained in another study [29].

Zakharkin et al. Sources of variation in Affymetrix microarray experiments. BMC Bioinformatics 2005, 6:214

Here is another paper that reports batch effects consistent with what you have reported:

In the SMRI brain expression microarray data set, batch effects accounted for nearly 50% of the observed variation in expression, to which site effects contributed 42% and date effects 7.3%.

Chen C, Grennan K, Badner J, Zhang D, Gershon E, et al. (2011) Removing Batch Effects in Analysis of Expression Microarray Data: An Evaluation of Six Batch Adjustment Methods. PLoS ONE 6(2): e17238. doi:10.1371/journal.pone.0017238

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your answer, its very helpful. To answer your questions: both arrays came from the same 2 sets of subjects, moreover they came from the same RNA extracts, so they are technical replicates for hybridization, but not for RNA extraction. There were more differentially expressed genes in teh confounded experiment, but not by much. $\endgroup$ – slava Feb 6 '14 at 20:02
  • $\begingroup$ There were 5737 differentially expressed genes in confounded experiment and 5041 without confounding. 3360 genes overlap between both experiments. $\endgroup$ – slava Feb 6 '14 at 20:05
  • $\begingroup$ @slava See my edit, it looks like studying variation of microarray data is a field unto itself. Perhaps even if the first results are unreliable, comparing the technical replicates can be a paper on its own. $\endgroup$ – Livid Feb 6 '14 at 21:26
  • $\begingroup$ This a great answer, looks that I will have to do a whole lot more of data exploration and dig deeper into the literature. Thanks again for this very detailed answer. $\endgroup$ – slava Feb 7 '14 at 19:00
  • $\begingroup$ Great issues have been raised here. I would also caution against false negatives, and recommend using the bootstrap to get confidence intervals for the rank of each gene in differential expression. Such confidence intervals are multiplicity adjusted in every way, and point out vividly how difficult is the task. For example, putative "loser" genes will be found to be consistent with being "winners" much of the time. $\endgroup$ – Frank Harrell Mar 11 '14 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.