Anscombe suggested two symmetrizing transforms for the negative binomial with mean $m$ and known exponent $k$* in his 1948 paper[1]:
$\ln(y+\frac{k}{2})$; with variance approximately $\psi'(k)$. This transformation approximates the more accurate but more complicated one below -
$2\sinh^{-1}\sqrt{\frac{y+c}{k-2c}}$, for $k>2$ where if $k$ is large $c\approx\frac{3}{8}$
In this case variance is also approximately $\psi'(k)$. If $k$ is not small this is approximately $\frac{1}{k-\frac12}$, but the function $\psi'$ is often available (it is in R for example).
Anscombe later suggests that a slightly smaller $c$ might be more suitable, and suggests rounding $c$ to a single decimal place; this would suggest $c=0.3$ might be a reasonable choice.
Note that the approximation in 1. is obtained by setting $c=k/2$ in 2., which is not generally very close to the optimum but may be more convenient.
* Specifically: $\Pr(Y = r) = \frac{\Gamma(r+k)}{r! \, \Gamma(k)} \left(\frac{m}{k+m}\right)^r \left(\frac{k}{k+m}\right)^k \quad\text{for }r = 0, 1, 2, \dotsc$
This is the same as the pmf listed in the Wikipedia article on the negative binomial under Alternative formulations at the fifth bullet point ("In negative binomial regression, ..."), but with the roles of $r$ and $k$ interchanged (so here $k$ is the shape parameter and the variance of this NB is $m+\frac{m^2}{k}$. (I've written it this way to fit with Anscombe's notation.)
Anscombe begins the paper with the transformation I list second, and initially gives it without the factor of 2 out the front (which only impacts the variance). He adds the factor of 2 later (presumably because it enables him to deal with both transformations together in several parts).
Once standardized to a residual, the factor of 2 makes no difference, of course.
In each case, the usual Anscombe residual would be
$\qquad r^A_i=\frac{t(y_i)-t(\mu_i)}{\sqrt{\text{Var}(t(y_i))}}$
where $t$ is either the transformation in 1. or in 2., the Var term is replaced by its approximation above, and of course we would replace $\mu_i$ by its fitted value.
[1]: Anscombe, F. J. (1948),
"The Transformation of Poisson, Binomial and Negative-Binomial Data,"
Biometrika, Vol. 35, No. 3/4 (Dec.), pp. 246-254