For maximum-likelihood estimation, you'll need to solve the score equations numerically: see http://en.wikipedia.org/wiki/Negative_binomial_distribution#Maximum_likelihood_estimation. (Or directly maximize the log-likelihood.)
For method-of-moments estimation, following e.g. the parametrization given herehere, substitute the sample mean & variance for the population mean $\operatorname{E}Y$ & variance $\operatorname{Var}Y$, & solve for the parameters $\mu$ & $\theta$. In this case the estimates are
$$\tilde\mu = \bar y$$
$$ {\tilde\theta}= \frac{\bar y^2}{s_y^2 - \bar y} $$
where $\bar y$ is the sample mean, & $s_y^2$ the sample variance.