4
$\begingroup$

Possible Duplicate:
Testing hypothesis of no group differences

Suppose I have $k$ samples from 2 independent experiments (service times by 2 methods) and their means are similar. How do I statistically show that both methods have similar service times?

Improve this question
$\endgroup$
5
  • $\begingroup$ Are you only concerned with testing differences in means? or are you interested in other distributional differences? Also when you say "populations" do you mean "samples"? $\endgroup$
    – Jeromy Anglim
    Sep 22, 2011 at 5:36
  • $\begingroup$ No, the objective is to statistically show that the service offered by both methods take similar amounts of time. $\endgroup$
    – Jacob
    Sep 22, 2011 at 19:47
  • $\begingroup$ Okay. To phrase the question slightly differently, are you interested only in testing consistency of means? or are you also interested in consistency of other distributional properties such as variance or skew? $\endgroup$
    – Jeromy Anglim
    Sep 22, 2011 at 22:25
  • $\begingroup$ Your question is very similar to this one on testing an hypothesis of no group differences: stats.stackexchange.com/questions/3038/… $\endgroup$
    – Jeromy Anglim
    Sep 23, 2011 at 5:06
  • $\begingroup$ @JeromyAnglim: That's exactly what I was looking for $\endgroup$
    – Jacob
    Sep 23, 2011 at 12:25

3 Answers 3

Reset to default
2
$\begingroup$

You can use equivalence testing here. Since you are testing to see if the two population distributions are similar I think it is a mistake to base conclusions off of a two sample t-test.

Here is the approach. Take the ratio of the two service times and construct the CI for a given significance level.

Now, the hard part, you need to scientifically determine what cutoffs indicate they are the same (the cutoffs do not come from the data). For example the FDA might specify the 90% CI of the ratio must be in the range of .80 to 1.25 to consider two drugs equivalent.

If your entire CI is within the cutoffs then you can conclude the two distributions are similar. If part of the CI is in the cutoff and part outside the results are inconclusive. If the entire CI is outside the cutoffs then you can conclude they are different.

Improve this answer
$\endgroup$
1
$\begingroup$

You have two distributions of service times. What you want is to compare those distributions and check whether they are really different.

There are a few statistical tests that can do this for you each with different drawbacks (e.g. sensitivity to changes in scale, location, etc.)

Have a look at the Kolmogorov-Smirnov test or Mann–Whitney U

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ Trouble is they're hypothesis tests which can provide evidence that the distributions differ, but won't show that they are similar. A high p-value could be due to low power. $\endgroup$
    – onestop
    Sep 21, 2011 at 14:21
  • $\begingroup$ @onestop: I think you get my problem. My understanding is that hypothesis tests can tell if two distributions are statistically different but not if they are the same. $\endgroup$
    – Jacob
    Sep 21, 2011 at 14:24
  • 2
    $\begingroup$ @Jacob It is almost certain that the two distributions are different, so get over it! The crux of the matter is whether the difference is large enough to be of concern. That suggests additional information is needed; namely, the purpose of your determination. If you conclude both methods have "similar" service times, what then? If due to chance (and unknown to you) your conclusion is erroneous, what implications would that have? Your answers to these questions would indicate how to compare the two empirical distributions you have. $\endgroup$
    – whuber ♦
    Sep 21, 2011 at 14:57
  • $\begingroup$ I agree with whuber. Unless they're identical in every respect, they're different. What you want to know is how serious that difference is. Additionally, if you're going to have to rely on a low power test, that's something you should have accounted for in designing the experiment. $\endgroup$
    – Fomite
    Sep 21, 2011 at 17:10
  • $\begingroup$ @whuber: Thanks for weighing in! The claim is that if both methods have similar service times, then one method can be replaced by another without affecting service times. If the conclusion is erroneous, then service will take longer. $\endgroup$
    – Jacob
    Sep 21, 2011 at 18:26
0
$\begingroup$

Set a high alpha level and show that you fail to reject the null.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.