This is a great question. The short answer is, I think, what you suggested, i.e. typically we want to use an a/b test to infer something about a larger (super) population, say of potential users, that, theoretically is infinite.
The long answer is more complicated but more interesting.
TL;DR. For finite (rather than infinite) causal inference, observations in treatment and control group are not independent so while you could use the finite population correction you'd also have to include a covariance term but the covariance term generally can't be estimated. 1) Dropping the finite population correction is a conservative way to deal with this problem. 2) Under the assumption of the sharp null this can be justified. 3) The correlation coefficient can be used to justify a (generally) smaller estimate for variance.
General case
Define the population size as $n$ and any subset of the population as $n_t$, $n_c$, etc. Define $W_1, W_2, ... W_n$ where $W_i$ is a random variable that indicates inclusion ($W_i = 1$) or exclusion ($W_i = 0$) from the sample. Furthermore, define $y_1, y_2, ... y_n$ as our variable of interest. We denote $y$ with the lower-case because per the design-based approach to sampling (and to causal inference) $y$ is fixed.
For a simple random sample without replacement, the sample mean, $\frac{1}{n_t} \sum^n_{i = 1} W_i y_i$, is an unbiased estimator for the population mean. It can be shown that
$$
Var(\frac{1}{n_t} \sum^n_{i = 1} W_i y_i) = \frac{S_t^2}{n_t} \cdot (1 - \frac{n_t}{n})
$$
The second term, $(1 - \frac{n_t}{n})$, is referred to as the finite population correction. If you work through the derivation, you will see that it arises because, without loss of generality, in sampling without replacement the first unit is not independent of the second unit. (The first unit is selected with probability $\frac{n_t}{n}$ and the second unit is selected with probability $\frac{n_t - 1}{n -1}$ if $W_1 = 1$ and with probability $\frac{n_t}{n -1}$ if $W_1 = 0$.) Of course, as $n \to \infty$, $\frac{n_t}{n} \to 0$, and the correction disappears because the dependence disappears.
Now we know that
$$
Var(X - Y) = Var(X) + Var(Y) - 2 \cdot Cov(X, Y)
$$
and that when $X$ and $Y$ are independent the covariance term is zero. This is the case for causal inference for a infinite (super) population.
Causal inference for finite populations
Causal inference for a finite population is a special case.
Define $t_1, t_2, ... , t_n$ and $c_1, c_2, ... , c_3$ as the value for the variable of interest for a unit if it is treated or not treated respectively (or assigned to treatment a or b). We denote both vectors with the lower-case because, from the perspective of the potential outcome framework, these values are fixed.
We'd like to recover the average treatment effect (difference in means)
$$
\frac{1}{n} \sum^n_{i=1} t_i - \frac{1}{n} \sum^n_{i=1} c_i
$$
but this is impossible because we have to assign some units to treatment (and for those units we don't observe the value of $c$) and other units to control (and for those units we don't observe the value of $t$).
This random assignment is identical to the mechanism of random sampling defined above. Thus, we estimate the average treatment effect as
$$
\bar{t} - \bar{c} = \frac{1}{n_t} \sum^{n}_{i=1} W_i t_i - \frac{1}{n_c} \sum^{n}_{i=1} (1 - W_i) c_i
$$
but different assignments will (generally) produce different estimates.
$$
\begin{aligned}
Var(\bar{t} - \bar{c}) &= Var( \frac{1}{n_t} \sum^{n}_{i=1} W_i t_i ) + Var(\frac{1}{n_c} \sum^{n}_{i=1} (1 - W_i) c_i) - 2 \cdot Cov(\frac{1}{n_t} \sum^{n}_{i=1} W_i t_i, \frac{1}{n_c} \sum^{n}_{i=1} (1 - W_i) c_i) \\
\end{aligned}
$$
We have the first and second term from above. It can be shown that the covariance is $-\frac{1 }{n (n - 1)}\sum_{i=1}^n (t_i - \bar{t}) \cdot (c_i - \bar{c})$. Making these substitutions gives us
$$
\begin{aligned}
Var(\bar{t} - \bar{c}) &= \frac{S_t^2}{n_t} \cdot (1 - \frac{n_t}{n}) + \frac{S_c^2}{n_c} \cdot (1 - \frac{n_c}{n}) + \frac{2 }{n (n - 1)}\sum_{i=1}^n (t_i - \bar{t}) \cdot (c_i - \bar{c})
\end{aligned}
$$
The problem is that while we can estimate the first and second term, we generally have no way of estimating the last term because we never observe $t_i$ and $c_i$ together rather only $t_i$ but not $c_i$ or vice versa.
The most common approach is to drop the finite population correction and the covariance. The resulting estimator for the variance is then $\frac{S_t^2}{n_t} + \frac{S_c^2}{n_c}$ which is generally conservative.
Variance for sharp null
Alternatively, this approach can be justified under the sharp null hypothesis of no treatment effect on any unit by using the variance of the unit-level treatment effects which we denote by $S_{ct}^2$.
$$
\begin{aligned}
S_{ct}^2 &= \frac{1}{n-1} \sum_{i=1}^n [ (t_i - \bar{t}) - (c_i - \bar{c})]^2 \\
&= \frac{1}{n-1} \sum_{i=1}^n [ t_i - \bar{t} ]^2 + \frac{1}{n-1} \sum_{i=1}^n [ c_i - \bar{c}]^2 - \frac{2}{n-1} \sum_{i=1}^n (t_i - \bar{t}) (c_i - \bar{c}) \\
&= S_t^2 + S_c^2 - \frac{2}{n-1} \sum_{i=1}^n (t_i - \bar{t}) (c_i - \bar{c})
\end{aligned}
$$
which means that $\frac{2}{n-1} \sum_{i=1}^n (t_i - \bar{t}) (c_i - \bar{c}) = S_t^2 + S_c^2 - S_{ct}^2$. Using this substitution for the equation above and the fact that $n_t = n - n_c$ and $n_c = n - n_t$ we simplify further but first note that
$$
\frac{S_t^2}{n_t} \cdot (1 - \frac{n_t}{n}) = \frac{S_t^2n - S_t^2n_t}{n n_t} = \frac{S_t^2n - S_t^2(n - n_c)}{n n_t} = \frac{S_t^2n_c}{n n_t}
$$
and likewise for $\frac{S_c^2}{n_c} \cdot (1 - \frac{n_c}{n})$. This gives us
$$
\begin{aligned}
Var(\bar{t} - \bar{c}) &= \frac{n_c S^2_t}{n n_t} + \frac{n_t S^2_c}{n n_c} + \frac{1}{n}(S_t^2 + S_c^2 - s_{ct}^2) \\
&= \frac{n_c S^2_t}{n (n - n_c)} + \frac{S^2_t}{n} \frac{(n - n_c)}{(n - n_c)} + \frac{n_t S^2_c}{n (n - n_t)} + \frac{S_c^2}{n} \frac{(n - n_t)}{(n - n_t)} - \frac{s_{ct}^2}{n} \\
&= \frac{n_c S^2_t + S_t^2 n - S_t^2 n_c}{n (n - n_c)} + \frac{n_t S^2_c + S_c^2 n - S_c^2 n_t}{n (n - n_t)} + - \frac{s_{ct}^2}{n} \\
&= \frac{S^2_t}{n_t} + \frac{S^2_c}{n_c} - \frac{S_{tc}^2}{n}
\end{aligned}
$$
Returning to the definition of $S^2_{ct}$ note that if a treatment has no effect for each unit, then $(t_i - \bar{t}) - (c_i - \bar{c}) = 0$ so the last term will equal zero.
Bounds on variance
Another approach is to utilize the correlation coefficent. Because the correlation coefficient is bound between $[-1, 1]$ we can establish an upper bound on the variance by substituting $\rho_{ct} = 1$.
$$
\begin{aligned}
Var(\bar{t} - \bar{c} | rho = 1) &= \frac{n_c S^2_t}{n n_t} + \frac{n_t S^2_c}{n n_c} + \frac{2}{n(n-1)} \sum_{i=1}^n (t_i - \bar{t}) (c_i - \bar{c}) \\
&= \frac{n_c S^2_t}{n n_t} + \frac{n_t S^2_c}{n n_c} + \frac{2}{n (n-1) s_t s_c} \sum_{i=1}^n (t_i - \bar{t}) (c_i - \bar{c}) \cdot s_t \cdot s_c \\
&= \frac{n_c S^2_t}{n n_t} + \frac{n_t S^2_c}{n n_c} + \frac{2}{n} \rho_{ct} \cdot s_t \cdot s_c \\
&= \frac{n_c S^2_t}{n n_t} + \frac{n_t S^2_c}{n n_c} + \frac{2 (1)}{n} \cdot s_t \cdot s_c \\
\end{aligned}
$$
This approach typically produces a smaller estimate for variance than $\frac{S^2_t}{n_t} + \frac{S^2_c}{n_c}$. However, the difference is usually marginal.
The result for finite population causal inference is due to Neymar. For an English translation see of the key paper here. For more on bounds for the variance see "Sharp bounds on the variance in randomized experiments," by Aronow, Green, and Lee. For a thorough introduction to this discussion see Causal Inference by Imbens and Rubin.
