Skip to main content
1 of 4
B.Liu
  • 1.4k
  • 10
  • 20

Note: The question was subjected to multiple round of edits, when other answers were made in between. This answer is made after Edit 2 was posted.


In the world of experiment design involving $t$-tests, one will need to have a rough idea on what the following five things may be:

  • The desired significance level ($\alpha$)
  • The desired test power ($\pi_{\min}$)
  • The effect size (in real terms, $\theta = \textrm{consumption}_{\textrm{after}} - \textrm{consumption}_{\textrm{before}}$)
  • The spread of the responses ($\sigma^2$ - it can be the pooled variance); and
  • The sample size ($n$)

In practice, given the (rough) formula for determining minimum sample size assuming normality assumptions and/or CLT practically apply [1]:

$$n_{\min} = \left(\frac{z_{1-\alpha} - z_{1-\pi_{\min}}}{\theta}\right)^2 \sigma^2,$$

where $z_{q}$ is the $q$th quantile of a standard normal, if you specify four of the five quantities above, you are basically constrained on the one left. Usually, $\alpha$ and $\pi_{\min}$ is assumed to be of certain value (0.05 and 0.8 in my field), and you mentioned the sample size is more or less fixed. This leaves the effect size and the spread as unknowns.

You then ask:

Since I have no idea how much will the water consumption mean will be in the end how can I do this? Maybe there are some pilot studies where I can get a standard deviation estimation, but is that enough?

which suggest to me that it is easier for you to estimate the variance / standard deviation than the effect size. Furthermore, I (as a layman in water technologies) would imagine is it easier to control how much water a device can save on average than how spread out the water savings are.

Thus, if you can get a standard deviation estimate, the formula will be able to tell you what effect size you will need to obtain a statistically significant result. (A side note that here my effect size is in real terms, i.e. average number of litres the water device can save a day, instead of Cohen's d, which is quoted in your question.) I personally will try and vary the estimate in both directions a bit and see how that affects the effect size.

This leads back to your key question:

Is a sample size of 50 enough for the paired t-tests?

Look at the effect size that comes out from above - is that a realistic amount of water your machine can save on average? If so, yes.

If not, i.e. you are expecting a smaller effect size, you might need to consider:

  1. Having more samples (which you said is pretty much constrained);
  2. Settling for a lower test power (i.e. have a lower chance to see a significant result if there is indeed a saving);
  3. Choosing a lower significance level (i.e. reject H_0 when p<0.1 instead of 0.05, risking more false positives); or
  4. Praying and hoping the test subjects' water usage behaviour (and hence the water savings) are more consistent, reducing the spread of the responses.

All of the above are just ways to balance the system / equation showing the relationship between the five quantities. The key takeaway is that sample size is not the only consideration when it comes to designing experiments, though it is often the most easily manipulatable parameter.


[1] From background material of one of my previous work (Section 3) - unfortunately I was unable to get it out quick enough and hence it remains as a pre-print: https://arxiv.org/pdf/1803.06258.pdf

B.Liu
  • 1.4k
  • 10
  • 20