Suppose I have a large population and I want to test if installing a new special light bulb can reduce energy consumption. Since I have a large population, I'll have people that usually consume high and some that consume less energy on a daily basis; therefore I would like to know the impact of the new special light bulb depending on their previous behavior. So I want to cluster my population into groups based on their energy consumption. Suppose that I get K groups after clustering. How can I determine the minimum sample size for each of my K+1 groups (+1 because I'm counting with the group that doesn't have the new light bulb, i.e, control group) to ensure some statistical rigor (for example some confidence level, etc..). I've only encountered literature about stratified sampling and cluster sampling, but none of those are what I want to do. In stratified sampling, they divide the population into K groups and then take elements from all of the groups to make sure every group will have elements in the (only) sample. In clustering sampling, they divide the population into K groups and only consider some of those groups for the sample.

EDIT 1: I found about one-way ANOVA: it can determine sample sizes for multiple groups but assumes as a null hypothesis that all of the groups have equal means (which is not the case here). In this case, the null hypothesis would have to be something like "there's no changing on the mean (energy consumption) on any of the groups with the new special light bulb". Even if one-way ANOVA could be used, I would have to give as an input a guess/estimation of the means for each group after getting the new special light bulb?

EDIT 2: I'd like to thank StatsStudent for his answer, which is very detailed and informative. However, his answer is not what I'm actually looking for. To be more detailed, I have the data on the energy consumption of the whole population (without the new special light bulb). And after clustering, I know there are K different types of behavior (for example, K=3: low, medium, and high energy consumption behaviors among the population). I want to know if installing the light bulb will affect the energy consumption of any of the K groups. Also, one aspect that would also like to include is to have more than one special light bulb, for example, three different light bulbs: A, B, and C. Can I use Multi-Way ANOVA to determine the sample size needed? I've read that this test is an extension of ANOVA, where there is more than one category, but only one category is of interest. The other category/categories are things that need to be controlled for (blocking/nesting/random effects/etc.). If I'm interested in determining if there is a difference in energy consumption due to light bulbs (A, B, and C) while controlling for energy behavior (low, medium, high) (K=3:groups from clustering) can and should I use two-way ANOVA? This will result in $3 \times 3 = 9$ groups.

  • $\begingroup$ I'm a little confused by your description and hope you can clarify. Is each household measured before the introduction of the new bulb and then you introduce the new bulb to every household at the same time and then want to determine if there is a reduction in energy usage after it's introduction? Or is only some subset of households getting the new bulb? $\endgroup$ Commented Dec 3, 2020 at 9:56
  • $\begingroup$ If it's not too late, and you haven't conducted the experiment yet, I'd suggest a simple paired sample t-test in which you measure, for say, over the course of a year the energy usage by each household. Then introduce the bulb to every household and for another year measure energy usage. Then each Household serves as it's own control which will reduce your variance and make your test more powerful. You'd simply take the difference in energy usage before and after the intervention of the bulb is introduced. Then carry out a simple t-test on the differences. Of course (continued) $\endgroup$ Commented Dec 3, 2020 at 10:02
  • $\begingroup$ Each household is measured BEFORE the introduction of the new light bulb. I have that data and I use clustering to get K groups from it. Now I want to know the minimum sample size for the K+1 groups (including the control group) to see if there is a statistically significant difference, i.e, I want to know if the new light bulb really changed the energy consumption in any of the groups. $\endgroup$
    – Numbermind
    Commented Dec 3, 2020 at 10:04
  • $\begingroup$ this design would suffer from potential time effects (e.g. other, external factors during the second year could lead to the reduction in energy usage or even weather patterns. So one could modify this design with a regression model and half of the households could randomly receive the new bulb treatment first and the other receive it in the second measurement period. $\endgroup$ Commented Dec 3, 2020 at 10:04
  • $\begingroup$ OK. There's really no need to do any clustering. Use each household as it's own cluster in some sense. I'll explain in an answer more fully below. $\endgroup$ Commented Dec 3, 2020 at 10:05

2 Answers 2


You essentially have a before and after experiment in which each participating household is provided the new bulb at a point in time and measurements are taken on energy usage before the "intervention" of the bulb is introduced and then additional measurements are taken for the same amount of time after the bulb is introduced. In this case, you'll want to treat each household as its own cluster really and perform a paired-sample t-test (there are more complex designs that aim at reducing bias, but this is a good starting point):

Under the paired sample $t$-test, each household is measured for say, 1 year. After 1 year, the bulb is introduced and for the next year energy consumption is measured. Denote the energy consumption of household $i$ before the bulb is installed in time-period 1 as $Y_{i,1}$ and the energy consumption of the household $i$ after the bulb is installed at time-period 2 as $Y_{i,2}$, for $i=1, 2, ..., n$, where $n$ is the total number of participating households.

Then you'll compute the difference between the two time periods for household $i$ as:


You'll then compute the simple average of of these $n$ differences. Call this $\bar{Y}_{diff}$. In addition, you'll compute the variance as $SD_Y^2={{\sum_{i=1}^n(Y_{i,diff}-\bar{Y}_{diff})^2}\over{n-1}}$ and compute the standard deviation as $SD_Y=\sqrt{SD_Y^2}$.

You will then want to test the Null Hypothesis, $H_0$, that the mean difference between time periods is equal to zero vs. the Alternative Hypothesis, $H_1$, that the mean difference between period 2 and 1 is greater than 0 (you're expecting the bulb introduced in time period 2 will result in less energy than in time period 1). In other words, you'll want to test:

$H_0: \mu_{diff}=0$ vs $H_1: \mu_{diff}>0$.

To test this hypothesis, You'll compare the test statistic: $T={\bar{Y}_{diff}\over{SD_Y/\sqrt{n}}}$ with $n-1$ degrees of freedom to the standard $t$-distribution with $n-1$ degrees of freedom. To perform this test at the 99% confidence level for example, you'd compare the value of your test statistic $T$ with $t_{.01}$ with $n-1$ degrees of freedom. If $T >t_{.01, n-1}$, you'd reject the null hypothesis and conclude that there is sufficient evidence at the 99% confidence level to conclude there is a reduction in the mean energy consumption.

Now, the question remains how large of a sample do you need? The answer depends on a number of factors. It's helpful to know what your budget is in planning this type of experiment because often time budgets dictate how much you can spend so sometimes you may be advised to sample as many households are you budget allows. Disregarding cost for the moment, you should try to select a sample size that provides you with sufficient power to detect true differences in mean energy consumption when they exist. Many statisticians select 80% power, which means there is an 80% chance of rejecting the null hypothesis when the alternative hypothesis is true (i.e. there is reduced energy consumption from the new bulb). Then you'll have to make some educated guesses on how much energy, on average the homes actually consume in time periods 1 and 2 (call these $\mu_1$ and $\mu_2$). You might be able to get estimates from your existing data and experiments with your bulb connected to a energy meter in the lab for example. You'll also need to determine an estimate of the standard deviation of the differences in energy consumption $\sigma_Y$. You'll then use these parameters for determining the sample size, $n$. The easiest way to compute this is to use any of the available online sample size calculators. I'll use this one here: https://www.stat.ubc.ca/~rollin/stats/ssize/n2.html. There is a reference on the web page to the formula used to compute this if you are interested in that.

So for example, let's assume $\mu_1=52$ and after introducing the bulb you expect the mean energy consumption to fall to $\mu_2=50$ and there is a standard deviation of $4$. Let's also say you'll want to test your hypothesis at the 99% confidence level. Then you simply input these parameters into screen as shown below and click calculate:

enter image description here

After doing that, you'll see that the sample size required is 63. You'll then need to sample 63 households. Keep in mind that this assumes 100% participation and compliance with every home in your experiment. If you suspect 10% of the participants might drop out, you should inflate this sample size estimate by 10%.

That all being said, this test assumes that there are no confounding variables between the two time periods. This may not be the case. For example an extremely hot summer in time period 2 may mean that more households run their AC units more often in time period 2 than in time period 1 and this is actually what caused the increase in energy in time period 2, rather than the bulb. To get around this, I'd recommend using a regression model and randomly assign half the households to get the bulb in time 1 and the other half in time 2. Then you can take this into consideration in building a regression model to better control for these types of external factors, but this is beyond the scope of my answer.

You may also find this reference quite detailed and provide methods for computing sample sizes in R: https://stats.idre.ucla.edu/r/dae/power-analysis-for-paired-sample-t-test/


Since no one, besides StatsStudent, provided an answer, I guess I'll answer it myself. I think I understand it now.

If I'm interested in determining the minimum sample size needed for the mentioned example with three light bulbs $A, B$ and $C$ and with the $K$ behaviors obtained from clustering (let us assume for this example that $K=3$ and that we label it as $k_1,k_2$ and $k_3$), the Multi-way ANOVA should be considered, more specifically a two-way ANOVA. Why? Because we have two factors: one for the light bulb $\{A,B,C\}$ and another with behaviors $\{k_1,k_2,k_3\}$ and two-way ANOVA is an extension of the one-way ANOVA that examines the influence of two different categorical independent variables (these two factors previously mentioned) on one continuous dependent variable (energy consumption).

For this calculation, softwares like $\texttt{G*Power}$ or $\texttt{R}$ can be used. For the latter one, the function $\texttt{wp.kanova}$ from the $\texttt{WebPower}$ package can be used. To determinate the minimum sample size, one has to give for input: Numerator degrees of freedom, Effect size, Number of groups, Significance level, and Statistical power. All of these inputs are directly known except for the effect size. For the effect size, one needs a previous trial study to calculate it or we can simply assign a guess value for it (carefully). The $\texttt{R}$ function will return the total number of samples, that we will need to divide with the number of groups and, thus, we will have the number needed for each group (in this case there are $3 \times 3 =9$ groups).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.