6
$\begingroup$

I am a scientist and by no means a statistician. I have, I think, a very basic question for you all.

I have performed the same experiment twice (analyzing levels of a certain protein) and am comparing levels in untreated vs treated animals. I performed a t-test with welch correction (because there was significant variance). I am observing a similar trend, but neither experiment alone reaches statistical significance. Is there a way to combine the two experiments to increase power? Although the trend is similar, the actual values are different between the experiments, which is, I think, increasing my standard error. Also, if anybody knows how to perform this analysis using graphpad prism, that would be ideal.

Thanks so much in advance for any assistance you may offer. Please let me know if you need more information.

$\endgroup$
5
  • 3
    $\begingroup$ Would it be correct to assume your combined data could be accurately and completely represented by a collection of ordered triples of the form $(i, x, y)$ where $i$ identifies the experiment, $x$ indicates whether treatment was applied or not, and $y$ is the protein level? Note that this implies there exist no other physical relationships among the data: no two triples refer to the same animal, animals were not housed together, they aren't groups of siblings, and so on. $\endgroup$
    – whuber
    Commented Oct 1, 2013 at 21:54
  • 2
    $\begingroup$ In what sense were these experiments separate then? $\endgroup$
    – AdamO
    Commented Oct 1, 2013 at 21:59
  • 2
    $\begingroup$ "I performed a t-test with welch correction (because there was significant variance)" -- this is not an advisable approach. Unless you have - a priori - reason to think that the variances will be similar, it's better (in terms of your tests having close to their nominal properties) to just use the Welch procedure, rather than use a hypothesis test to decide when to use it. $\endgroup$
    – Glen_b
    Commented Oct 1, 2013 at 22:51
  • 2
    $\begingroup$ I think the critical thing is not that these are separate experiments but that t-tests have already been done. $\endgroup$
    – John
    Commented Oct 1, 2013 at 23:46
  • $\begingroup$ Thanks everyone! To clarify: I performed experiment 1 (E1) using 10 animals (5/group) and measured protein levels. I performed a t-test (using graphpad prism) which told me that the variance between groups was sign. different. I then re-analyzed, t-test + welch. I then repeated with 10 new animals. Neither experiment alone was sign. The trends are the same (treatment reduced protein). However, the levels in both groups (treated and untreated ) are higher in E1 than in E2. If I combine experiments, my error bars are huge. So, my question is, can I combine the data from both experiments? $\endgroup$
    – Brenda
    Commented Oct 2, 2013 at 16:20

1 Answer 1

3
$\begingroup$

Effects should be different across experiments, and so should variances. That's the nature of sampling. What you have is just different samples being different. There's no way to know which estimate of variance is closer to true value, or even guess at it with the information you've given and equal N's in the samples. So, while you'd like it to be the smaller one, that may not be correct. More than likely the average variance is best.

Your general tactic here of searching for an effect may eventually bare fruit. You may combine all of the subjects into one experiment, run a few more, drop an outlier here or there, look for various analysis techniques, and voila, a significant effect. Maybe you won't do all of that but I'm trying to point out that you're thinking about it wrong. Use the data you have to make your best determination about the truth of the matter, not show an effect.

An important thing to keep in mind is that an unstated assumption about any statistical test is that you're performing it because you want to know the answer to the test, not because you've previously done other tests and failed to find what you would like to find. So now, because you've already done the test, the rate of Type I error is no longer what you set it to be, alpha. You're increasing the probability of finding an effect whether there is one or not.

That said, you could do something that's not a test. You could construct a confidence interval of the effect through a mega-analysis (just combine all of the data) and report that as a higher quality estimate of the effect than either experiment had alone. You will have to concede that what you've done is post hoc and describe the tests that you did do already. But this is probably the best way to report what you've done so far.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.