TLDR: I need to determine the minimum sample size for measuring the energy consumption of code.
Hello, I'm trying to determine how many samples (N) I need in an experiment to be confident that my data's mean and standard deviation are representatives of the population. Note, I'm not well versed in statistics, hence why I'm struggling to find the correct formula to use.
The experiment: I'm measuring the energy consumption of running a piece of code repeatably. My data set will contain N amount of samples, where each sample is a continuous variable (energy in joules). However, I need to calculate how many N samples I should acquire. If necessary, I can do an initial experiment to get an estimation of the standard deviation, mean, and such. Then use that to calculate whether I have enough samples or I need to acquire more.
I have previously tried to use Cochran's formula. However, I used the Shapiro-Wilk test and found out that some of my data is not normally distributed (I actually have multiple data sets with different pieces of code and the same code with different measuring instruments, but I would like to use a formula that works on the non-normally distributed data). If my understanding is correct, that means I should not use Cochran's formula.
I can (within reason) acquire as many samples as I want and as such am not limited by the population size or the difficulty in getting more samples, I would just like to calculate the minimum required samples, as to be able to argue why some I picked to run my experiments N times.
Goal/Plan: I'm comparing measurement of the same code, but measured by different measuring instruments as to compare them. I'm probably going to use the Mann-Whitney U Test to test if the test case measurement are independent of each other. Then I'm going to do a non-parametric rank correlation (Kendall's Tau coefficient) to calculate the correlation between the different measuring instruments and also our ground truth.
Non-normal distribution: I don't know what specific distributions the non-normal distributions are. I have done a normalcy test and discovered that while some of the data sets are normally distributed, several are not.