Suppose you sample $n$ members from a population of $N$ (without replacement) and $k$ of them are failures. The definition of confidence tells us to ask this question:
If there are $K$ failures in the population, what is the chance we observe $k$ or fewer failures in the sample?
Without going into the combinatorial details, let's just call this number $p(k,K;n,N)$. It can be used to establish upper confidence limits on $K$ through a form of logical inversion. Let $N$ and $n$ both be known and $\alpha$ be a specified probability. If $K^\prime$ is so large that $p(k,K^\prime;n,N) \lt \alpha$, then it is unlikely we would have observed $k$ or fewer failures in the first place. This gives us confidence that the true number of failures, $K$, is strictly less than $K^\prime$.
Pushing this reasoning to its natural limit, we therefore seek the smallest value $K^\prime$ for which $p(k, K^\prime; n, N) \lt \alpha$. We will use $K^\prime - 1$ for the $1-\alpha$ upper confidence limit on $K$. Equivalently we could maximize the value $K^{\prime\prime}$ for which $p(k, K^{\prime\prime}; n, N) \ge \alpha$:
$$\operatorname{UCL}_\alpha(k) = \max\{K\,|\, p(k, K; n, N) \ge \alpha\}.\tag{1}$$
Now for the details. The chance of observing exactly $k$ failures is the chance that (a) our $n$-element sample contains those $k$ failures and (b) the remaining $N-n$ members of the population contain the remaining $K-k$ failures. This describes $\binom{K}{k} \binom{N-K}{n-k}$ subsets out of $\binom{N}{n}$ equally likely subsets. Summing these for all values from $k=0$ to $k$ equal to the actual number of observed failures gives
$$p(k,K;n,N) = \frac{1}{\binom{N}{n}}\sum_{j=0}^k \binom{K}{j} \binom{N-K}{n-j}.$$
This is the Hypergeometric distribution.
In R, for instance, the parameters to supply to the Hypergeometric functions are $N-K$ (called m on the manual page), $K$, (n), and $n$ (k). The phyper function implements $p$ and the qhyper function implements its inverse.
Take, for example, a case of a population with $N=8$ elements from which a sample of size $n=4$ is drawn and $k=1$ failure is observed. Then
$$p(3, K, 4, 8) = \frac{1}{\binom{8}{4}}\sum_{j=0}^1 \binom{K}{j}\binom{8-K}{4-j} = \frac{1}{70}\left(\binom{8-K}{4} + K\binom{8-K}{3}\right).$$
The possible values of $K$ range from a minimum of $k=1$ (the one failure observed) to $k = k + (N-n) = 5$ (occurring when every non-observed member of the population is a failure). Plugging these values into the preceding equation gives the sequence
$$(70, 55, 35, 17, 5)/70 \approx (100, 79, 50, 24, 7)/100.$$
R will compute them in one stroke as
phyper(1, 1:5, 8-(1:5), 4)
We read these numbers like this:
There is $100\%$ confidence the population has at least $K=1$ failure. (We have seen it.)
There is $79\%$ confidence the population has at least $K=2$ failures. In other words, we attach high confidence to the existence of at least one more failure in the $N-n=4$ members not observed.
There is $50\%$ confidence the population has at least $K=3$ failures. This may seem counter-intuitive: since we have seen half the population and observed $k=1$ failure, shouldn't we assign exactly $1/2=50\%$ confidence to seeing one more failure in the other half of the population? This is where confidence differs from probability. The correct approach asks this question: when there are $K=3$ failures in the population (of size $N=8$) and we sample half of it, what is the chance we will see just zero or one failures? By symmetry--the unsampled members themselves constitute a random sample of $N-n=4$, too--this is the chance that the remaining unsampled members will consist of just zero or one failures. Thus observing zero or one failures out of three in the population is an event that will occur half the time. Consequently, actually observing one failure is perfectly consistent with the presence of three failures total.
There is $24\%$ confidence the population has at least $K=4$ failures and $7\%$ confidence it has at least $K=5$ failures. These numbers are starting to get close to typical values of $\alpha$. For instance, with $\alpha=0.10$ the $90\%$ upper confidence limit for $K$ would be $K=4$. But with $\alpha=0.05$ the upper $95\%$ UCL for $K$ is $K=5$. When we observe one failure out of four, in a sample from a population of eight, there is an appreciable risk that all the unsampled members are failures! This is because when five of eight members are failures, there's still a considerable chance--more than $7\%$--that our sample just happens to include all three successes.
Note that qhyper in R does not compute confidence limits. You need to search, just as we did in this example. A brute-force search (but relatively efficient for R) tests all the values, as in
which(phyper(1, 1:5, 8-(1:5), 4) >= .10)
This command returns the indexes 1 2 3 4, showing that the first four elements of the vector 1:5 (representing the possible values of $K$) are consistent with our observations at the $\alpha=0.10$ level. The largest of those, $4$, corresponds to $K=4$ as we found through inspection.
In the example of the question, a sample of size $n=100$ is taken from a population of $N=500$ and $k=3$ failures are observed. What is a $90\%$ upper confidence limit for the total number of failures $K$? The R search is
`max(which(phyper(3, 1:100, 500-(1:100), 100) >= .10))`
(The correspondence between this and the mathematical formula for the UCL in $(1)$ is evident.)
It returns a UCL of $30$. Let's double-check by computing the probabilities $p(3,30;100,500)$ and $p(3,31;100,500)$. The first should exceed $10\%$ and the second should drop just below it:
> phyper(3, 30, 500-30, 100)
[1] 0.1151626
> phyper(3, 31, 500-31, 100)
[1] 0.09959309
That's exactly what happens. We conclude, with at least $90\%$ confidence, that there exist up to (but no more than) $K-k=30-3=27$ additional failures among the $N-n=500-100=400$ unsampled members of the population.
