Let $X_1, \dots, X_n$ be iid with cdf $F$. Let $\hat{F}(x) = \frac{\sum I(X_i \leq x)}{n}$ be the empirical distribution function. Suppose $x < y$ and compute $Cov(\hat{F}(x),\hat{F}(y))$. This is what I have done but my computations don't give me the correct answer.
I'll use $Cov(X,Y) = E(XY) - E(X)E(Y)$. We know $E(\hat{F}(x)) = F(x)$ so the $E(X)E(Y)$ part is easy. Next we have $E(\hat{F}(x)\hat{F}(y)) = E(\sum_{i,j}I(X_i \leq x)I(X_j \leq y)) = \sum_{i,j}E(I(X_i \leq x)I(X_j \leq y))$. I will split this sum up into two pieces, the one with $i=j$ and the rest.
For the $i=j$ part we have $E(I(X_i \leq x)I(X_j \leq y)) = E(I(X_i \leq x)I(X_i \leq y)) = F(x)$ since $x = min(x,y)$
The other case ($i \not= j$) is easy because the expectation of the product two independent variables is the product of expectations and hence $E(I(X_i \leq x)I(X_j \leq y)) = F(x)F(y)$.
Thus $Cov(X,Y) = E(XY) - E(X)E(Y) = E(X)E(Y) - F(x)F(y) = n\times F(x) + n\times(n-1)F(x)F(y) - F(x)F(y)$
which is incorrect.