How do you show that two populations are statistically similar? [duplicate]

Possible Duplicate:
Testing hypothesis of no group differences

Suppose I have $k$ samples from 2 independent experiments (service times by 2 methods) and their means are similar. How do I statistically show that both methods have similar service times?

• Are you only concerned with testing differences in means? or are you interested in other distributional differences? Also when you say "populations" do you mean "samples"? – Jeromy Anglim Sep 22 '11 at 5:36
• No, the objective is to statistically show that the service offered by both methods take similar amounts of time. – Jacob Sep 22 '11 at 19:47
• Okay. To phrase the question slightly differently, are you interested only in testing consistency of means? or are you also interested in consistency of other distributional properties such as variance or skew? – Jeromy Anglim Sep 22 '11 at 22:25
• Your question is very similar to this one on testing an hypothesis of no group differences: stats.stackexchange.com/questions/3038/… – Jeromy Anglim Sep 23 '11 at 5:06
• @JeromyAnglim: That's exactly what I was looking for – Jacob Sep 23 '11 at 12:25

You can use equivalence testing here. Since you are testing to see if the two population distributions are similar I think it is a mistake to base conclusions off of a two sample t-test.

Here is the approach. Take the ratio of the two service times and construct the CI for a given significance level.

Now, the hard part, you need to scientifically determine what cutoffs indicate they are the same (the cutoffs do not come from the data). For example the FDA might specify the 90% CI of the ratio must be in the range of .80 to 1.25 to consider two drugs equivalent.

If your entire CI is within the cutoffs then you can conclude the two distributions are similar. If part of the CI is in the cutoff and part outside the results are inconclusive. If the entire CI is outside the cutoffs then you can conclude they are different.

You have two distributions of service times. What you want is to compare those distributions and check whether they are really different.

There are a few statistical tests that can do this for you each with different drawbacks (e.g. sensitivity to changes in scale, location, etc.)

Have a look at the Kolmogorov-Smirnov test or Mann–Whitney U

• Trouble is they're hypothesis tests which can provide evidence that the distributions differ, but won't show that they are similar. A high p-value could be due to low power. – onestop Sep 21 '11 at 14:21
• @onestop: I think you get my problem. My understanding is that hypothesis tests can tell if two distributions are statistically different but not if they are the same. – Jacob Sep 21 '11 at 14:24
• @Jacob It is almost certain that the two distributions are different, so get over it! The crux of the matter is whether the difference is large enough to be of concern. That suggests additional information is needed; namely, the purpose of your determination. If you conclude both methods have "similar" service times, what then? If due to chance (and unknown to you) your conclusion is erroneous, what implications would that have? Your answers to these questions would indicate how to compare the two empirical distributions you have. – whuber Sep 21 '11 at 14:57
• I agree with whuber. Unless they're identical in every respect, they're different. What you want to know is how serious that difference is. Additionally, if you're going to have to rely on a low power test, that's something you should have accounted for in designing the experiment. – Fomite Sep 21 '11 at 17:10
• @whuber: Thanks for weighing in! The claim is that if both methods have similar service times, then one method can be replaced by another without affecting service times. If the conclusion is erroneous, then service will take longer. – Jacob Sep 21 '11 at 18:26

Set a high alpha level and show that you fail to reject the null.