In a scientific study there are 8 guinea pigs, 5 if which are pregnant. If 3 are selected at random without replacement, find the probability that are all pregnant?
And the answer is 5/28.
But when finding the probability that none are pregnant, the answer is 1/56 since (3/8)(2/7)(1/6) = 1/56.
Why can't we use complement upon solving this? And subtract 1 from 5/28?
There a 5 guinea pigs that are pregnant. On the first draw, you have a 5/8 chance of selecting a pregnant one. What do you think the chances are on the second draw?
For your part of the question "why can't I use the complement rule?". You can, but it's not as simple as $P(\bar{A}) = 1- P(A)$. What you have (1/56) is the probability of drawing NO pregnant guinea pigs. The complement of "drawing NO pregnant guinea pigs" is "at least one guinea pig is pregnant".
Short answer: The probability of getting 3 pregnant guinea pigs when 3 are chosen at random from the 8, is $$\frac{{5 \choose 3}{3 \choose 0}}{{8\choose 3}} = \frac{5}{28} = 0.1786.$$ Computations in R:
choose(5,3)/choose(8,3)
[1] 0.1785714
5/28
[1] 0.1785714
More detail: Let $X$ be the number of pregnant guinea pigs in a sample of 3 chosen at random without replacement from the 8 in the study. Then $X$ has a hypergeometric distribution with PDF:
$$P(X = k) = \frac{{5 \choose k}{3 \choose 3-k}}{{8\choose 3}},$$
for $k = 0,1,2,3.$ We can use the PDF function dhyper in R to make
the distribution table.
k = 0:3; pdf = dhyper(k, 5,3, 3)
cbind(k, pdf)
k pdf
[1,] 0 0.01785714
[2,] 1 0.26785714
[3,] 2 0.53571429
[4,] 3 0.17857143
About complements: The reason you can't take the complement as you propose, is that the complement of "no (0) pregnant guinea pigs" is not "all (3) pregnant guinea pigs." There are other possible outcomes: specifically, either 1 or 2 pregnant. The complement of "no pregnant guinea pigs" in the sample is "at least one pregnant guinea pig."
$$P(X \ge 1) = 1 - P(X = 0).$$
Bar graph of PDF: Here is a graph of the hypergeometric PDF of $X.$
