I am working on a project where I am evaluating software packages. I have got the results of two experiments where only one variable has been changed. Each experiment has been conducted 10 times. I am doing paired-sample t-tests. For some of my metrics, the results are exactly the same for the ten iterations. Therefore I have a standard error of zero and I cannot do the t-test.

If I am correct you can use the p-value generated during a t-test to support your claim that the different values of the variables had an impact on the results (as you can say that your results are statically significant). In my case, what can I say/do to support such claim?


EDIT: From the comment, I am going to add details on my design.

I'm am evaluating file carvers, these are digital forensics tools which can be used to extract files from forensics images (such as hard drives). I've planted a known number of evidence (500) and configured each file carver to look for these evidence. I have got tow set of evidence. One is a "well known" evidence (that I created myself for the purpose of the research, so I am planting 500 time the same file) and a realistic set of evidence (composed of 500 unique files). I am doing two tests with the same tool: one for each set of planted evidence. If I understood correctly the different type of t-tests, I can do a paired t-test?

From the results that I have got, the tools fail each time to recover all evidence. However, the recovered evidence are the same each time. The tools are working as a search function. Such function might fail if it is run hundreds of times, maybe I need much more. I have tested four tools with different scenarios and the results never varied (same true positives, false positives and false negatives)

What I need, is a way to support my findings. When you look at the literature you get told how to talk about p-values results. I just have no idea what I should say if it cannot be computed as their is no variation in my results. I found some articles where the researchers had similar results but they were just ignored (only displayed in the table). It might be obvious but as it is the first time that I'm using stats in my work I'm a bit lost (and I'm self-taught so I might have poorly understood some concepts)

Finally, my main question is: The results show clearly that the type of evidence as an impact on the results. How can I support such claim in a scientific fashion?

  • $\begingroup$ Please tell us more about what you are doing. It seems to me that, in software evaluation, a low variance might be ideal; we want computers to perform consistently. But what is it you are doing? $\endgroup$ – Peter Flom Apr 12 '12 at 0:36
  • 2
    $\begingroup$ Are you sure that you need a statistical test? If the variance is negligible then you can just look at the difference between the means and declare a winner. If there is no difference between the means then there is no difference... $\endgroup$ – Michael Lew Apr 12 '12 at 0:53
  • $\begingroup$ I do no know if I need a statistical test. Actually, I do not care if I have to use one or not. I just do no know how I can support my claims scientifically as the literature that I've got mostly focus on stats. $\endgroup$ – Flanfl Apr 12 '12 at 8:28
  • $\begingroup$ Do you have any external knowledge that suggests the different tools should either recover exactly the same things (e.g. they use the same library as their workhorse) or that that actually should not be the case? I once got exactly equal results in a completely different setting but where I knew the results should be slightly differing. In the end it lead me to discover a rather severe bug... $\endgroup$ – cbeleites unhappy with SX May 12 '12 at 13:52

How about a permuted or randomized t-test? That would let you avoid computing the variance entirely. They make minimal assumptions about the data and are pretty easy to perform.

If you're unfamiliar with them, the idea is actually pretty simple. Start by calculating the difference between the two groups' means. We'll call that the "observed difference." Under the null hypothesis, the treatments have the same/no effect, so the group labels are meaningless. To test this, randomize or shuffle the group labels and compute the difference between the means of these "fake" groups. Repeat this process many times to generate a distribution of "shuffled differences." Finally, assign a p value by asking how often one sees a shuffled difference at least as extreme as the observed one: $$p=\frac{\textrm{#(Shuffled > Observed)}}{\textrm{# Shuffled}+1}$$ You may need to take the absolute value for a two-tailed test.

I think you should consider rolando2 and Michael Lew's questions carefully too. In particular:

  • Is your design actually paired? To review, a paired t-test is usually used when the same subjects are examined in both conditions (e.g., give everyone drug A, measure, wait and give everyone drug B and measure again), while the unpaired t-test is typically used when there are two non-overlapping conditions (half the subjects get drug A, half get drug B). There's not much in your description that suggests that you've got "paired data." Perhaps you could elaborate a little bit on your design?

  • Do you have sufficient power? If you're looking for relatively rare effects, they might not show up in your 10 sample/condition data set. If you've got a rough idea of expected difference between the means, you could do a power analysis to see if you have enough samples.

| cite | improve this answer | |
  • $\begingroup$ I've added details in my question. If you need more feel free to ask. $\endgroup$ – Flanfl Apr 12 '12 at 8:28

Suppose you looked at it in quality control terms. Will you, or will your audience for this research, really be satisfied to learn that no failure has occurred after 10 trials? (I'm guessing that that is an accurate way to describe your situation.) You may have to use a much larger sample for each of your 2 groups if failures are going to be rare. And then a paired t-test would not be applicable: you'd probably want to use a test of the difference between dependent proportions, if your 2 groups are really paired. On the other hand, I could imagine a situation in which 10 trials per group is all you can realistically generate, in which case you'd report the descriptive results you obtained without running any significance test.

| cite | improve this answer | |
  • $\begingroup$ I've added details in my question. If you need more feel free to ask. $\endgroup$ – Flanfl Apr 12 '12 at 8:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.