0
$\begingroup$

TLDR: I need to determine the minimum sample size for measuring the energy consumption of code.

Hello, I'm trying to determine how many samples (N) I need in an experiment to be confident that my data's mean and standard deviation are representatives of the population. Note, I'm not well versed in statistics, hence why I'm struggling to find the correct formula to use.

The experiment: I'm measuring the energy consumption of running a piece of code repeatably. My data set will contain N amount of samples, where each sample is a continuous variable (energy in joules). However, I need to calculate how many N samples I should acquire. If necessary, I can do an initial experiment to get an estimation of the standard deviation, mean, and such. Then use that to calculate whether I have enough samples or I need to acquire more.

I have previously tried to use Cochran's formula. However, I used the Shapiro-Wilk test and found out that some of my data is not normally distributed (I actually have multiple data sets with different pieces of code and the same code with different measuring instruments, but I would like to use a formula that works on the non-normally distributed data). If my understanding is correct, that means I should not use Cochran's formula.

I can (within reason) acquire as many samples as I want and as such am not limited by the population size or the difficulty in getting more samples, I would just like to calculate the minimum required samples, as to be able to argue why some I picked to run my experiments N times.

Goal/Plan: I'm comparing measurement of the same code, but measured by different measuring instruments as to compare them. I'm probably going to use the Mann-Whitney U Test to test if the test case measurement are independent of each other. Then I'm going to do a non-parametric rank correlation (Kendall's Tau coefficient) to calculate the correlation between the different measuring instruments and also our ground truth.

Non-normal distribution: I don't know what specific distributions the non-normal distributions are. I have done a normalcy test and discovered that while some of the data sets are normally distributed, several are not.

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ "and found out that some of my data is not normally distributed", what does this "some" mean? $\endgroup$
    – Firebug
    Mar 2 at 13:40
  • $\begingroup$ I have several experiments testing different code, i.e. several data sets. Some of the data sets are not normally distributed, while some of them are. However, I would like to use the same sample size formula on all the data sets if possible. Hope that cleared it up. $\endgroup$
    – Jamie Baldwin Pedersen
    Mar 2 at 13:43
  • 1
    $\begingroup$ why not just run your experiment until you have your answers to a required level of precision? A sample size calculation requires that you specify that precision needed anyway, so since you don't have any practical limits I can't see the value in working it out beforehand. $\endgroup$
    – George Savva
    Mar 2 at 13:59
  • $\begingroup$ The idea is to run an initial experiment for each setup, and using the initial data to calculate how many more samples are needed. Then use the maximum amount of needed samples for all the experiments. I wanted to this because I'm going to run a lot of automated experiments and I want to be able to argue why I chose to run the experiments N times. $\endgroup$
    – Jamie Baldwin Pedersen
    Mar 2 at 14:08
  • $\begingroup$ It would help a lot if you could describe what your analysis is likely to be, and how you will interpret the data. Also what you mean by 'not normally distributed'. When we do sample size calculations we are trying to anticipate your analysis and predict what the uncertainty in your estimates is going to be. So the information about the analysis plan and the distribution of the pilot data is important. If you're worried about distributional assumptions you can run a sample size calc by simulation/bootstrapping. But as I said above I'm not sure you need one at all. $\endgroup$
    – George Savva
    Mar 2 at 16:14

1 Answer 1

Reset to default
0
$\begingroup$

Firstly, on the general statistical question, "what sample size do I need to estimate the mean of an unknown distribution to a given level of certainty?". This has no answer. In fact, some distributions don't even have a mean, e.g. the Cauchy distribution. Most statistics, or data modelling, depends on prior knowledge.

Secondly, on distributions of code running time. There is a lot of prior knowledge available to you. If you want to look at a complex case, read about the performance of the JVM. While often it is very good, it is also very unpredictable. It depends on whether and when the JIT kicks in, and also one whether and when the garbage collector kicks in.

Even at a low level there are complexities, e.g. on the state of the cache lines.

So you need to do some data modelling, appropriate to the actual code that interests you, before you can decide the sample sizes.

Improve this answer
$\endgroup$
3
  • $\begingroup$ In the case of study design, we can also replace "prior knowledge" with reasonable or risk minimizing assumptions, For instance, estimating a proportion, the widest SEs and CIs occur when p=0.5, so this can be a starting place for investigation. $\endgroup$
    – AdamO
    Mar 2 at 17:40
  • $\begingroup$ I'm not sure about what the distribution is, I have only done a normalcy test and discovered that some of the sample groups are not normally distributed. Regarding the second point, I'm not measuring run time, I'm measuring energy consumption. There is limited data on the subject of measuring energy consumption, and most of the papers don't reason why they decided to run their experiment N times. I have seen some examples of papers basing their sample size on the central limit theorem and running about 30 times. $\endgroup$
    – Jamie Baldwin Pedersen
    Mar 3 at 8:22
  • $\begingroup$ You can find lots of papers that assume normalcy. Sometimes the central limit theorem is relevant - but sadly often, it is the convenience of the investigator which drives that assumption. I am not surprised that some of your samples are non-normal. Unless your runs are long, the central limit theorem won't kick in. $\endgroup$
    – chrishmorris
    Apr 26 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.