1
$\begingroup$

I have the following problem:

I recorded the execution time of a program 30 times, then implemented an optimization and recorded the execution time 30 times again (on the same machine). I am not sure if I should use an unpaired t-test or a paired t-test to prove that the optimization made a significant difference in the execution time.

On the one hand, the distributions are not independent and it doesn't matter which "individuals" are paired; on the other hand, pairing these measurements makes little sense to me because there are no meaningful "individuals" that could be paired.

The closest example for which a paired t-test was supposed to be used was an experiment where the paired individuals shared one characteristic while some other condition changed (e.g. two individuals with the same IQ), however I don't know which characteristic would justify pairing two measurements in my problem together.

Any help would be greatly appreciated!

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Looks unpaired to me. Is there some particular feature of your set-up which makes you doubt that? $\endgroup$
    – mdewey
    Commented Aug 14 at 14:43
  • $\begingroup$ @mdewey The feature that stay the same is the computer and the program stays similar, but other than that I don't know. I often found 'before-and-after' comparisons to be used as an example for when paired t-tests should be used which made me question which test I am supposed to use. $\endgroup$
    – Alwin Zomotor
    Commented Aug 14 at 14:52

1 Answer 1

Reset to default
4
$\begingroup$

This appears to be an unpaired problem - if I understand correctly, you're running the same program 30 times in a row on the same test bed, not running it over a battery of 30 different tests. The observed runtimes of repeated executions of a computer program are likely independent and identically distributed, unless there is some effect where the runtime will actually depend on the run order - this could happen if there's a memory leak, for example, making later executions take longer, or if there is a series of different tests of varying difficulty.

Those possibilities aside, it doesn't sound like there is any meaningful order to the 30 observations, and no meaningful way to pair between observations in the first set and observations in the second set. If "it doesn't matter which individuals are paired", then you should not be running a paired test - your result will depend entirely on the arbitrary pairing you happened to use, which has no particular justification over any other pairing.

Improve this answer
$\endgroup$
3
  • $\begingroup$ @AlwinZomotor It greatly depends on the pairing, and you can't even run a paired t-test without equal sample sizes (as some observations have no pair). The paired test allows you to control for some of the variability between observations, by using each individual as its own baseline. An example is to observe some fixed cohort before and after some intervention, and compare each individual to their own baseline. It makes sense to compare Person A's pre- and post-intervention scores, but it doesn't make sense to compare Person A's pre-intervention score to Person B's post-intervention score. $\endgroup$
    – Nuclear Hoagie
    Commented Aug 14 at 15:26
  • $\begingroup$ @AlwinZomotor I struggle to imagine in what sense you could consider observations "paired" if you could actually just pick any two random samples and assign pairs at whim. "Paired" implies there is some kind of correspondence or relationship between the samples, you can't pair samples arbitrarily. $\endgroup$
    – Nuclear Hoagie
    Commented Aug 14 at 15:30
  • $\begingroup$ You are right; I was misunderstanding the test in that regard. $\endgroup$
    – Alwin Zomotor
    Commented Aug 14 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.