Besides simulations, there is an approximate (but generally quite good) solution and a more accurate solution. Not all libraries or software offer the more accurate solution; you'll have to refer to their documentation to know where they stand. All of this is assuming an equal number of observations in the two groups1.
There are software and libraries that make it very easy to use the more accurate solution. I give an example in R at the end of this answer. However, if you're stuck with a tool that does not offer this solution, you could use the approximate one, but then it may be safer to double check the result with a simulation.
This approximation requires using the pooled standard deviation as a denominator of the effect size (Cohen's d), as suggested by Jacob Cohen (1988):
$$\sigma' = \sqrt{\frac{\sigma_1^2+\sigma_2^2}{2}}$$
where $\sigma_1$ and $\sigma_2$ are the standard deviations of the two groups.
For instance, in the pwr library in R, if you want to detect a difference of means of 1, with a standard deviation of 0.5 in the first group, and 1.5 in the second group, with a power of 0.8, and an alpha level of 0.001, here is what you could do:
library(pwr)
sd1=0.5
sd2=1.5
delta=1
pooled.sd = sqrt((sd1^2 + sd2^2)/2)
pw = pwr.t.test(power=0.8, d = delta/pooled.sd, sig.level = 0.001,
alternative="two.sided")
pw$n #45.41933
If we round up the result, it gives a required sample size of 46 observations per group.
This solution is just an approximation though, and if you run a simulation with the sample size it suggests, you may find that you get a power (a bit) under what you originally required. Here is an example in R, where a simulation of 10,000 repeated experiments shows that with a Welch's t-test we have in fact a power of about 0.785 (instead of 0.8), with the previously suggested sample size of 46 observations per group:
set.seed(0)
results = c()
for (i in 1:10000){
a = rnorm(n=ceiling(pw$n), mean=0, sd=sd1)
b = rnorm(n=ceiling(pw$n), mean=delta, sd=sd2)
res = t.test(a, b, alternative = "two.sided", var.equal = FALSE)
results = c(results , res$p.value)
}
mean(results<0.001) # 0.7851
If this is really a problem to you, it shouldn't be too difficult to find how many observations you need to add to this suggested sample size, in order to reach the power you want2.
Now, where does this discrepancy between the pwr library and simulations come from? And how to get a more accurate estimation of the required sample size in the first place?
The problem is that power calculations for the Welch t-test should involve changing the number of degrees of freedom too, not just the denominator of the effect size. That's why in the example above, we end up with a power that is a bit different of what it should be, because under the hood the pwr library calculates the number of degrees of freedom as:
$$\nu= (n_1-1) + (n_2-1)$$
when it should be using3 :
$$\nu = \frac{\left( {\sigma_1^2 \over n_1} + {\sigma_2^2 \over n_2} \right)^2 }
{ \frac{\sigma_1^4}{n_1^2 (n_1-1)} + \frac{\sigma_2^4}{n_2^2 (n_2-1) }}$$
There are software and libraries that do compute correctly the number of degrees of freedom for the case of unequal variances, giving you a more reliable estimation of the required sample size. In R, this is for example the case of the MKpower library. Using the same scenario as before, here it is:
library(MKpower)
sd1=0.5
sd2=1.5
delta=1
pw = power.welch.t.test(n = NULL, delta = delta, sd1 = sd1,
sd2 = sd2, sig.level = 0.001, power = 0.8,
alternative = "two.sided")
pw$n #47.12973
It suggests 48 observations per group, when the pwr library suggested 46 observations. Let check with a simulation how accurate this is:
set.seed(0)
results = c()
for (i in 1:10000){
a = rnorm(n=48, mean=0, sd=sd1)
b = rnorm(n=48, mean=delta, sd=sd2)
res = t.test(a, b, alternative = "two.sided", var.equal = FALSE)
results = c(results , res$p.value)
}
mean(results<0.001) # 0.8136
48 observations per group give you a power of about 0.81, a bit above what you required. If you tried the simulation with 47 observations per group, you'll find that the power goes slightly under 0.8. So if you want at least a power of 0.8, this is really 48 observations per group you need.
In general, just to be safe, it may be a good idea to use simulations to double check the results this kind of tool gives you4.
Footnotes
1 For the case of unequal variances and unequal sample sizes, see this other thread: Calculate statistical power on groups with different size
2 This should not be a lot. I've yet to see a case where it requires more than 2 or 3 additional observations per group, but perhaps there are specific scenarios where it could be substantially more.
3 One implication of the pwr library not being able to change the degrees of freedom for something more relevant is that, in general, you should refer to the documentation of the tool you're using. How does it take into account the case of unequal variances? If the documentation does not clarify it, ask their support or user forums.
4 Why not using only simulations in the first place to determine the required sample size, without resorting at all to these libraries? Because if can save you some substantial computation time if you calculate power or required sample size for a large variety of different scenarios, e.g. if you use power curves.
Reference
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Routledge. https://doi.org/10.4324/9780203771587.
