# Ideal test/p-value calculation for difference in means with small sample size and right skewed data? [closed]

I need to compare a response variable between two different groups. My control group has 205 observations while my comparison group only has 2 observations (unfortunately this is the nature of the data I'm working with and I can't get any more). What methods might be appropriate to get a p value for difference in means? The response variable has a right skew and is continuous between 0 and 1. I'm looking at different ways to deal with this and I fear that one group only having 2 observations limits my options. It rules out a trimmed mean test, and resampling has a high variance.

• I ask a question (or several related questions) in my answer, responses to which may be necessary to give much in the way of useful help. Since it might be easy to miss (and may be better answered up here rather than a comment under my answer), I will highlight it here. "What is this variable measuring? Is it an angle (e.g. as a fraction of a circle), a proportion (and if so, of what on what)" Commented Jun 21 at 14:53
• What sort of measurement are these two samples from the control group? Are these like two measurements in a single person or other type of unit? In that case you would be interested in variability of a measurement within a single person. The variability of the 205 observations, if that are measurements without repetitions within the same units/persons will not give you a good indication of the type of variability within a single person.... Commented Jun 23 at 18:15
• ... in addition, it might be doubtful what it would mean if the means are different for such situation. The group with 2 samples is a different type of distribution (the mean value of a person plus/minus some error) from the distribution of the group with 205 samples (the value of means occuring in a population). In this case nearly always you will have that the means are different. For example, say that we perform some taste test for an apple tree and the score is calculated to be between 0 and 1, then most apples trees will have an average that is different from the mean. Commented Jun 23 at 18:17
• That is just one out of many potential interpretations. Your question is not sufficiently clear/precise. Commented Jun 23 at 18:21

Honestly, trying to do statistics on a sample of size 2 is like being "up the creek, w/o a paddle". This is not, and never will be, a representative sample of any population, and any inference you can make will not be credible.

About the only thing I would do is look at the data. Plot the control sample (histogram, or even individual value plot, with an indication of the mean), and its CI (do not worry about the skew: 205 is sufficiently large, and the t-test is robust to non-extreme departures from normality; the CI will be close enough). Then plot the 2 data points for your comparison group (different color), and see where they land.
In fact, if you added such a plot to your question, the answer may just "hit you between the eyes".

• "This is not, and never will be, a representative sample of any population, and any inference you can make will not be credible." I'd suggest it depends on the size of the population. If your population is five and you sample two, that's not so bad. Commented Jun 23 at 18:45
• If the population size is five, then measure all 5, and forego any statistics. Commented Jun 23 at 21:28
• That's not always feasible. For example, you could imagine a stratified sample with many strata. The populations in some strata might be quite small, but it may still not be efficient to fully enumerate (rather than sample) the small strata. Commented Jun 23 at 22:12

The limiting factor is the smaller of the two sample sizes. Because the power is exceptionally low in your situation, it is misleading to use p-values, as large p-values have no interpretations. If you really need to do statistical inference, stick with a confidence interval. To handle skewness you might use the Hodges-Lehmann estimate of the difference between groups (which is concordant with the Wilcoxon test).

Ideal test/p-value calculation for difference in means with small sample size and right skewed data?

There's really no "ideal". Normally you want two things:

1. a significance level close to the nominal one (e.g. typically people would want the true level not straying more than a small amount above it nor too far below it)

2. Relatively good power against alternatives of interest.

If you want to focus more on the second thing, you will usually need a distributional model, at least a reasonable one.

You can get the first on a test of means nonparametrically (not with a rank based test), such as via a permutation test. As long as you have the sample size for it.

I need to compare a response variable between two different groups. My control group has 205 observations while my comparison group only has 2 observations (unfortunately this is the nature of the data I'm working with and I can't get any more).

Well, give up on

i) asymptotic results (the properties will be primarily be driven by that of the smaller sample; consider what would happen as the larger sample size went to infinity -- you'd end up with a one-sample test with $$n=2$$.)

ii) nonparametric tests. Neither permutation tests nor bootstrap tests will do you much good; the first cannot yield a test with any p-values below a typical $$\alpha$$, the second relies on large sample results. To get useful small sample results you'll need a model.

iii) having much power

You will want to use every possible piece of understanding of the variable, external information and expert knowledge you can garner.

I'm looking at different ways to deal with this and I fear that one group only having 2 observations limits my options. It rules out a trimmed mean test, and resampling has a high variance.

Trimmed mean would be ruled out anyway, even at large samples, unless you added assumptions that don't make sense to me. Consider a variable whose spread is related to the mean (as would generally be typical for variables with support on the unit interval); a difference of trimmed means won't normally be consistent for the difference of population means.

You can't do much about high variance, you have n=2.

The response variable has a right skew and is continuous between 0 and 1.

Well, that rules out location-shift alternatives. You will need to formulate carefully what your set of alternatives should be.

What is this variable measuring? Is it an angle (e.g. as a fraction of a circle), a proportion (and if so, of what on what), etc etc.

What methods might be appropriate to get a p value for difference in means?

You need a model. There's really nothing else for it.

You should not be in the position of scrambling for this sort of choice after you collect data. This stuff comes up front.

I would look at how many of your control group are greater than both the comparison group observations / how many are less than both. Which one to use as your statistic depends on which tail you're testing. If you're doing a two-tailed test, you should take whichever p-value is smaller, and then double it. It's not going to have much power, but it doesn't depend on any assumptions about the underlying distribution.