5
$\begingroup$

I have two datasets with 35 points each. I know that both are pulled from normal distributions with equal variances. A two-sample t-test indicates they have the same mean, but the p-value is close to my alpha.

  1. Knowing they are normal with equal variances, would it make sense to just create normal distributions with the relative sample means, draw say 1,000 points from each, and then perform a two-sample t-test?

  2. Would it be fruitful to do this, ie could I expect to learn anything new?

$\endgroup$
  • 2
    $\begingroup$ A nonsignificant result in a two-sample t-test does not indicate they have the same mean. It indicates that there is not enough evidence to rule out the possibility that they have the same mean. $\endgroup$ – brendan Oct 22 at 15:57
  • $\begingroup$ I look forward to less and less wieldy sentences as my understanding grows $\endgroup$ – user12686018 Oct 25 at 1:46
  • $\begingroup$ It is definitely one of the risks of most technical fields! $\endgroup$ – brendan Oct 25 at 9:20
9
$\begingroup$

No, you will assure yourself of eventually rejecting the null hypothesis of equality for a large enough sample size (1000 ought to do the trick unless the difference between sample means is tiny tiny tiny). All this would be doing is confirming your observation that the sample means are different, which you already know.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ This answer is spot on, but I want to add a bit more -- the measurement of the mean has uncertainty associated with it. This is the uncertainty that is taken into account by the t-test. If you resample from normal distributions using the sample means and sample variance (i.e., calculated from your data) then you assuming that these are in fact the the true means and variance. If you knew the true values, then you wouldn't need a statistical test. $\endgroup$ – brendan Oct 22 at 16:04
5
$\begingroup$

But...why? Your data is as ideal as could be. It satisfies nearly every assumption of sophomore stats. People only write about this kind of problem.

Resampling opens you up to simulation noise in which you could falsely reject/fail to reject simply because of simulation error. The statistical significance would not itself be significant.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Those first two words are important. Why do you want to do this, just to get from $p=0.51$ to $p=0.49?$ $\endgroup$ – Dave Oct 22 at 1:11
  • $\begingroup$ The motivating fear is that I don't have enough points to trust the answers, but if that's not an issue I'm happy to let it be $\endgroup$ – user12686018 Oct 22 at 1:38
  • 3
    $\begingroup$ I think in this case less number of points should not create a 'trust' problem because the assumptions of t-test are thoroughly valid. Small sample size would have been an issue if test was based on the asymptomatic distribution of the sample statistic. Here we are using the exact sampling distribution. $\endgroup$ – Dayne Oct 22 at 1:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.