Let's look at the simpler case of positive univariate random variables. I assume it's already clear that in general for a positive random variable whose variance is not $0$, that $\text{Cov}(X,1/X)< 0$. If you're happy to accept that as the case, then
$\text{Cov}(X,1/X) = E(X.1/X) - E(X)E(1/X) = 1-E(X)E(1/X) <0\,,$ so
$E(X) E(1/X) > 1\,,$ or
$E(1/X)>1/E(X)$
(or or you could just use Jensen's inequality to the same end).
(In fact we can show $1/E(X)<E(1/X)\leq 2/E(X)$, a rather neat little fact)
Generally we should not expect means to "match up" in the way you anticipated.