I would like to compare means/medians of two samples that may be very skewed in size, e.g. 25 points and 1 or 2 points and test them for similarity, rather than difference. I realize that the power of such a test is likely to be low, however I would like to calculate it nevertheless. I have been reading about equivalence tests but so far haven't seen much on the non-parametric side. Because of the small sample size, if there are ideas for permutation/simulation/bootstrap tests for similarity, I'd like to hear those too. Thanks much.
For the (still) interested R-user:
Assume you want to test at the 5% level the working hypothesis of "true shift D within -d and d", where d is the equivalence margin. Use the R-function "wilcox.test" to obtain a 90% c.i. for D (=median of the difference between a sample from group 1 and a sample from group 2). If this c.i. is contained entirely in [-d,d], then you could be 95% certain that your working hypothesis is true.
Remark: If you can show (somehow) distributional equivalence using e.g. 2-sample-KS-test, equivalence in any sort of location parameter follows.
Non-parametric tests such as the ones developed by Wilcoxon would be appropriate for such a task.
To analyze goodness of fit between two large sets you can consider: non-parametric tests such as Anderson-Darling or Kolmogorov-Smirnov. Anderson-Darling is pretty good for running samples of different sizes (if your samples get larger than just 1 or 2), in particular. HTH.
I found lots of relevant information in a paper and subsequent book by Stefan Wellek:
Unfortunately, neither of these sources are free. The book is supposed to have SAS/R code associated with it but the provided link is broken. The text does provide enough info to implement it though.