**NOTE** this answer precedes the OP's note about percentages. This answer assumed:

* 1/41 is the prior probability of infection in the tested population (not the population as a whole)

* 2/1000 is the false negative rate of the PCR test $P(PCR^{-}|Infected)$ 

----

I may be confused myself as my calculations do give probabilities summing to 1. 

> The analytical false positive rate of a PCR test is 2 in 1,000

I assumed you mean *false negative rate*, right? Otherwise how can you have 2.4% true positives (1/41), 6.8% positive tests but only 0.2% false positives?

Here's my reasoning. First let's set up a simple tree with 100k people (there is some rounding here):

```
                      100000 people
                           |
                 --------------------
                 |                  | 
              Infected          Not infected 
              P = 1/41           P = 40/41  
               2439                97561
                 |                   |
         -------------          ------------
         |           |          |          |
       PCR+        PCR-        PCR+       PCR-
     P = 0.998   P = 0.002   P = 0.044   P = 0.956
       2434          5         4366       93195
```

Here `P = 0.044` is the false positive rate and comes from `0.068 - (0.998 * 1/41)`.

And here's the probabilities:

$$
P(inf^{-}|PCR^{+}) = \frac{P(inf^{-}) \cdot P(PCR^{+}|inf^{-})}{P(inf^{-}) \cdot P(PCR^{+}|inf^{-}) + P(inf^{+}) \cdot P(PCR^{+}|inf^{+})} \\
= \frac{(40/41) \cdot 0.044}{(40/41) \cdot 0.044 + (1/41) \cdot 0.998} = 0.638
$$

and

$$
P(inf^{+}|PCR^{+}) = \frac{P(inf^{+}) \cdot P(PCR^{+}|inf^{+})}{P(inf^{-}) \cdot P(PCR^{+}|inf^{-}) + P(inf^{+}) \cdot P(PCR^{+}|inf^{+})} \\
= \frac{(1/41) \cdot 0.998}{(40/41) \cdot 0.044 + (1/41) \cdot 0.998} = 0.362
$$

If I'm correct, your mistake is in using 0.068 as denominator in the Bayes formulas.