Minimum sample size required in paired t-tests and statistic significance

Question

I want to test if new technology machines can make people consume less water in their homes.

I have around 150 of those machines, where they can be divided into 3 types of technologies (A, B, C), i.e. 50 of each.

I would like to perform a paired t-test on the 150 individuals, i.e, measuring their mean water consumption before and after the machine installation.

I thought about making 3 individual paired t-tests, one for each technology to test each technology. But also a paired t-test with all of the 150 (without taking into account the type).

However, I want to make sure that in the end, I have statistically significant results. I want to have a confidence interval and margin of errors in the end. Is a sample size of 50 enough for the paired t-tests?

I know I have to assume that the mean differences will follow a normal distribution. I've searched about getting the minimum sample size and saw that we need to give as input an estimation of the standard deviation of the mean difference and also the difference in population means.

Since I have no idea how much will the water consumption mean will be in the end how can I do this? Maybe there are some pilot studies where I can get a standard deviation estimation, but is that enough?

I saw this post What is a minimum sample size for a paired t-test and what is a non-parametric equivalent if data is non-normal?, but my questions remains.

Also, if the normal distribution assumption, in the end, turns out to be false, can I turn to Wilcoxon signed-rank in the end?

Another question that is bothering me is why are sample sizes in paired t-tests much lower when comparing to tests like two-way ANOVA (for example)? I see paired t-tests of size 30 while two-way ANOVA (with a control group) is around >200?

Edit 1: Should I conduct a paired t-test like I described or should I make an ANOVA test with a control group (with 150 individuals) and my test group (with 150 individuals)? For both of them, since I only got 150 machines, I guess my sample size is predefined, but how can I ensure that my tests will have significance, i.e, 95% confidence and a certain margin of errors?

Edit 2: Do I have to take into account the effect size or the power of the test? I've read that if I have a pilot study with a few individuals (e.g 8) where the study says the consumption before and after (paired) the installation of those machines, I can calculate the effect size with

$$\text{Effect size} = (\text{Mean}_{H1}-\text{Mean}_{H0})/\text{SD}_{pooled}$$

and then I can proceed to software R, for example, to determinate the sample size. For example, if effect size=0.47, significance level= 0.05 and power of 80%, I would get:

 pwr.t.test (d=0.47, sig.level =0.05, power=0.80, type="paired", alternative="greater")

which returns $n=29.39 \approx 30$ pairs.

So I'm guessing it really depends on the effect size, and for that, I need a pilot study.

Answer 1

What is the minimum sample size depends on the question: "Minimum sample size to accomplish what?".

A paired-t test can be done on as few as 2 pairs if the only goal is to be able do some computations and get an answer (and you do not care about the quality of the answer).

If the question is how big a sample size do you need for the Central Limit Theorem to allow you to use normal based tests like the paired-t when the population is not normal? then this depends on how non-normal your population of differences is. Intro stats classes and text books use a rule of thumb with numbers like 30, but those are not really justified other than keeping things simple in an introductory class. In some cases 6 is big enough, in other cases 10,000 is not big enough. An important thing to remember is that for the paired test it is the amount of skewness/outliers in the differences, not the original values that will be important. This is one of the reasons for using paired tests.

One question that I don't see asked or answered in your description is how much time before and after are you going to measure water consumption for? I would expect much more normality and lower variability in the average daily usage for 3 months worth of data before and 3 months after compared to a single day before and after.

If your question is the minimum sample size to have a certain power to detect a given effect size, then this really depends on the effect size that you want to see and how much variation you expect (the standard deviation of the differences). If you have no idea what these may be then you need to either do some more research, talk with an expert, or do a pilot study of some sort (or better, all 3). Think about what effect size will be meaningful, a large enough study could show a reduction in water consumption of 1 table spoon, but I doubt many people would care about that small of a change.

If you can get some information on current water consumption, then one approach to explore some of these issues is to simulate some data based on the data you can get and some assumptions about what may change (try different effect sizes, etc.), then analyze your simulated data to see if it gives you meaningful results (confidence intervals are precise enough to be useful, power, etc.).

Another issue to think about is seasonality of when you collect your data. Do the households in the area of interest consume different amounts of water during different seasons (if water used to water the lawn/garden is included in your measurements then this is probably a strong yes). If your before time points may differ significantly from your after time points in weather/temperature/etc. then you should make an effort to address this in your experimental design and analysis. One option would be to include another 50 "control" households that do not receive any device but are measured before and after to give an estimate of natural differences between before and after periods.

For the analysis, you can do paired-t tests, but it would probably be better to do a randomized block ANOVA design, or mixed-effects model (or a Bayesian Hierarchical model) with households as the blocks/random effects to still give you the pairing but also allow you to compare between the different machines (and control) and look at other factors.

You also ask why paired tests require smaller sample sizes than unpaired. Simply put, the sample size depends a lot on the amount of residual variation (variability after accounting for other factors), if pairing is natural, then it will also reduce that residual variation. In your case you are going to have household to household variation (a family of 4 will likely consume much more water than a single person), in a non-paired study that household to household variation will be included in the residual standard deviation, but proper pairing will remove/adjust for most of the household to household variation, so the paired sample size calculation will be based on a much smaller standard deviation than the unpaired equivalent.

Answer 2

According to what I have learned there is no minimum sample size for a t-test. In fact the t-test is suitable for cases where the $n$ sample size is: 3 and more.

_{Even $n=2$ would work.}

A paired t-test on observations $\{X_{1i}\}_{i=1}^n$ and $\{X_{2i}\}_{i=1}^n$ is the same as a one-sample t test on differences. *

You choose t-test well, since you don't know the $\sigma$ of the population, i.e. z-test would not work for your case unless you somehow interact with the God and get $\sigma$ from it.

In other words, you should do t-test as you do, where t-distribution is a sample distribution behind the sampling. This distribution further assumes you relay on sample SD $S$ (standard deviation of the sample).

Since $S$ brings uncertainty, unless $n$ is big (where we usually assume $\sigma \approx S$) we decrease the degrees of freedom.

$$ \frac{\bar{x}-\mu}{\sigma / \sqrt{n}} \sim z \longrightarrow \frac{\bar{x}-\mu}{s / \sqrt{n}} \sim t_{\mathrm{n}-1} $$

Once we calculate the sample mean $\bar X$ we can estimate the confidence interval.

$$ \bar X \pm t \frac{S}{\sqrt{n}} $$

Where $t$ you can get in R via the 95% confidence interval rule:

t = qt(0.975,df=n-1)

If the normal distribution assumption, in the end, turns out to be false, can I turn to Wilcoxon signed-rank in the end?

You need to have normal distribution or something like close to normal if your number of samples is relatively small say smaller than 30.

Someone said I should not intimidate with the 30 number, but for now I assume if I have at least 30 samples samples adhere to the Normal distribution based on the Central Limit Theorem.

I plan to calculate soon why the 30 is the important number in statistics, but I don't have the power to ask more questions at the moment :).

This will be possible based on KL divergence, but for now say that with $n=30$ we have the enough power to say the t-distribution is close to Normal.

To calculate the power I found this R code:

power.t.test(n = 20, delta = 1)
power.t.test(power = .90, delta = 1)

First one should answer the question what is the power of 20 samples, and second how many samples do you need to gain the 0.9 power.

_{I don't know what is delta in here, but it must me something important, documentation is lacking some detailed facts so I need to examine.}

So with 30+ samples you will have the normal distribution assumption, no need for rank tests.

why are sample sizes in paired t-tests much lower when comparing to tests like two-way ANOVA (for example)? I see paired t-tests of size 30 while two-way ANOVA (with a control group) is around >200

ANOVA simplified have to be the same as t-test but for 3 or more samples we compare. So if you have some strange results you may share the R or Python code to replicate.

---

Ref

Answer 3

Note: The question was subjected to multiple round of edits, when other answers were made in between. This answer is made after Edit 2 was posted, and refrained from dealing with the part on Wilcoxon and ANOVA as it is unlikely to add on what existing answers have.

---

In the world of experiment design involving $t$-tests, one will need to have a rough idea on what the following five things may be:

• The desired significance level ($\alpha$)
• The desired test power ($\pi_{\min}$)
• The effect size (in real terms, $\theta = \textrm{consumption}_{\textrm{after}} - \textrm{consumption}_{\textrm{before}}$)
• The spread of the responses ($\sigma^2$ - it can be the pooled variance); and
• The sample size ($n$)

In practice, given the (rough) formula for determining minimum sample size assuming normality assumptions and/or CLT practically apply [1]:

$$n_{\min} = \left(\frac{z_{1-\alpha} - z_{1-\pi_{\min}}}{\theta}\right)^2 \sigma^2,$$

where $z_{q}$ is the $q$th quantile of a standard normal, if you specify four of the five quantities above, you are basically constrained on the one left. Usually, $\alpha$ and $\pi_{\min}$ is assumed to be of certain value (0.05 and 0.8 in my field), and you mentioned the sample size is more or less fixed. This leaves the effect size and the spread as unknowns.

You then ask:

Since I have no idea how much will the water consumption mean will be in the end how can I do this? Maybe there are some pilot studies where I can get a standard deviation estimation, but is that enough?

which suggest to me that it is easier for you to estimate the variance / standard deviation than the effect size. Furthermore, I (as a layman in water technologies) would imagine is it easier to influence how much water a device can save on average than how spread out the water savings are.

Thus, if you can get a standard deviation estimate, the formula will be able to tell you what effect size you will need to obtain a statistically significant result. (A side note that here my effect size is in real terms, i.e. average number of litres the water device can save a day, instead of Cohen's d, which is quoted in your question.) I personally will try and vary the estimate in both directions a bit and see how that affects the effect size.

This leads back to your key question:

Is a sample size of 50 enough for the paired t-tests?

Look at the effect size that comes out from above - is that a realistic amount of water your machine can save on average? If so, yes.

If not, i.e. you are expecting a smaller effect size, you might need to consider:

1. Having more samples (which you said is pretty much constrained);
2. Settling for a lower test power (i.e. have a lower chance to see a significant result if there is indeed a saving);
3. Choosing a higher significance level (i.e. reject H_0 when p<0.1 instead of 0.05, risking more false positives); or
4. Praying and hoping the test subjects' water usage behaviour (and hence the water savings) are more consistent, reducing the spread of the responses.

All of the above are just ways to balance the system / equation showing the relationship between the five quantities. The key takeaway is that sample size is not the only consideration when it comes to designing experiments, though it is often the most easily manipulatable parameter.

---

[1] From background material of one of my previous work (Section 3) - unfortunately I was unable to get it out quick enough and hence it remains as a pre-print: https://arxiv.org/pdf/1803.06258.pdf