How can I maximize statistical power for small sample sets with a known, potentially non-normal, distribution? I want to estimate the mean difference of a variable between populations with some level of confidence and with a minimum number of samples collected. The populations share the same variance and distribution, which may look normal, bimodal, or whatever else. Right now, I use a t-test and collect 40 samples for each population to compensate for the fact that the distribution can be non-normal. I then look at the difference of means formula for t-tests to determine the mean difference's confidence interval. Typically this confidence interval is acceptably small. However, is there a way I can do this using fewer samples? Can I leverage the fact that I know the distribution precisely?
The specific use case is that I'm making a startup-time benchmarking tool that takes two versions of an application, the control version and the experimental version, and benchmarks each by recording their startup time $N$ times, and reports if the experimental version has a significantly slower or faster startup than that of the control. It has to be run quickly, meaning with as few samples as possible, to give users results in an acceptable amount of time. The application's variance and distribution changes very little over any 24-hour span, so every 24 hours I can get a large dataset for the latest version of the application, e.g. with 1,000 startup times, and use that for the next 24 hours for all versions of the application. However, over the past few years, the distribution has slowly gone between normal, bi-modal, and other sorts, so it's not very stable beyond 24 hours.
 A: Regardless of the distribution for the data generative process the difference in sample means is consistent and unbiased for the difference in population means.  Additionally, the two-sample Wald or t-test works for non-normal data so long as the sample means are approximately normally distributed.  Here is a related thread.
One way to investigate whether "approximately normal" holds in your particular setting would be to take bootstrap samples of size n (sample with replacement from your data set as if the data set is the population of interest) many times, each time calculating the difference in means.  If the resulting histogram of difference in sample means appears approximately normal then you can feel comfortable with the two-sample Wald or t-test.
If the histogram suggests the difference in sample means is not approximately normal you can use a link function (transformation) of the individual means or of the difference in means to improve the normal approximation.  One example is a log link.  This is not a transformation of the subject-level observations, it is a transformation of the sample means or the difference in means.
Using an appropriate non-identity link function would allow you to utilize a smaller sample size than you would otherwise when using an identity link.  Here is a thread discussing link functions when constructing a CI for a Bernoulli proportion.
