How do I calculate the sample size to compare two populations? I am working on a farming experiment and I have two batches of chickens that are growing in a side-by-side trial. The difference between them is how they are fed. I need to select some of the chickens from each batch to send in for nutritional analysis of their meat.
How do I know how many chickens from each batch to send into the lab so that the sample sent is indicative of the entire batch? I was told to look at and report p-value, but I am not really a statistics guy.
 A: This is a straightforward question of sample-size determination based on a power-analysis.
Essentially, you need to specify the alpha level you will use (typically 0.05), the power you wish (often 0.80), and a relevant effect size. Given these three input data and the assumption that you will calculate plain vanilla unpaired t tests, there is a number of sample size calculators online that will tell you the required $N$. If you have a more complex model, e.g., a regression with covariates, it is usually simplest to simulate.
What effect size to input into these calculations is usually the hardest decision. Often, people will look through the literature and use previously reported effect sizes, but because of publication bias, this (a) typically over-estimates the true population effect size, and (b) only accounts for statistical significance, not clinical significance. In other words, this approach may give you decent power to detect an effect size that is too large for the use that you want to put your analysis to, or too small.
The best recommendation is to use an effect size that you would be sorry to miss. For your poultry, changing the feed for all chickens in case your experiment comes out positive will likely entail some costs (either switchover costs, or the new feed may simply be more expensive), so you would require some minimum effect in the outcome, i.e., the nutritional value of the chicken. This minimum effect is the one you should power your analysis for.
Here is a very good earlier answer to a similar question.
If you do plan on using a complex model and are not a statistician, I recommend you think about retaining a statistical consultant. This may be cheaper than an overpowered study. And less frustrating than an underpowered one.
