# What is the distribution of $X$ since $X|Y \sim \mathrm{Bi}(Y, p)$ and $Y \sim \mathrm{NB}(r, q)$?

I have done the usual steps to calculate the marginal distribution of $X$, i.e. using the following formula:

$$p(X) = \int p(X,Y) dY = \int p(X|Y) p(Y) dY$$

Therefore, it holds: \begin{align} P(X=k) &= \sum_{N=0}^{\infty} P(X=k|Y=N) P(Y=N) \\ &= \sum_{N=k}^{\infty} \dbinom{N}{k} p^k (1-p)^{N-k}\dbinom{N+r-1}{N}(1-q)^r q^N \\ &= \left(\frac{p}{1-p} \right)^k (1-q)^r \sum_{N=k}^{\infty} \dbinom{N}{k}\dbinom{N+r-1}{N}(q(1-p))^N \end{align}

Can anyone analytically calculate the sum? I have succeeded only for $k=0$ using Newton's binomial series - http://en.wikipedia.org/wiki/Binomial_coefficient#Newton.27s_binomial_series

And if not, could anyone suggest how to approximate it?

It is possible to compute it exactly. From the last sum you wrote, change indexes defining $i=N-k$ ($N=k+i$). You get: $$\sum_{i=0}^{\infty}(q(1-p))^{k+i}\frac{(k+i)!}{i!k!}\frac{(k+r+i-1)!}{(r-1)!(k+i)!}$$
simplify the $(k+i)!$ and take out the sum the term $(q(1-p))^k$, it remains $$\sum_{i=0}^{\infty}(q(1-p))^i\frac{1}{k!(r-1)!}\frac{(k+r+i-1)!}{i!}$$ where I reordered the terms. Now the dependence on $i$ is on the last fraction. To get a binomial, we multiply and divide by $(k+r-1)!$: $$\sum_{i=0}^{\infty}(q(1-p))^i\frac{(k+r-1)!}{k!(r-1)!}\frac{(k+r+i-1)!}{(k+r-1)!i!}$$ now as binomials: $$\sum_{i=0}^{\infty}(q(1-p))^i{k+r-1\choose k}{k+r+i-1 \choose i}$$ and $${k+r-1\choose k}\sum_{i=0}^{\infty}(q(1-p))^i{k+r+i-1 \choose i}$$ the last sum result is $(1-q(1-p))^{-(k+r)}$, so the value of the original sum is $${k+r-1\choose k}(1-q(1-p))^{-(k+r)}(q(1-p))^k$$ and the final value of the probability you want is $$P(X=k)=(pq)^k(1-q)^r(1-q(1-p))^{-(k+r)}{k+r-1\choose k}$$