Method of Moments approach
Given a set of $n$ exponentially distributed i.i.d variables $X_i \sim EXP(1)$ the expected value of an ordered statistic $X_{i:n}$ is found in a straighforward fashion with the method of moments which gives the expected value as,
\begin{equation*}
\begin{aligned}[b]
E[X] = \left[\frac{\partial}{\partial t}\int e^{xt}f(x) \right]_{t=0}= \int xf(x)
\end{aligned}
\end{equation*}
Now, for the sake of rigor and clarity, consider the full pdf of the ordered statistic for a general integer $i ; 1<i< n$;
\begin{equation*}
\begin{aligned}[b]
f(x_i) &= \frac{n!}{(i-1)!(n-i)!}[F(x_i)]^{i-1}[1-F(x_i)]^{n-i}f(x_i) \\
&=\frac{n!}{(i-1)!(n-i)!}[1-e^{-x_i}]^{i-1}[e^{-x_i}]^{(n-i+1)x_i}
\end{aligned}
\end{equation*}
Applying the method of moments gives,
\begin{equation*}
\begin{aligned}[b]
E[X] &= \frac{n!}{(i-1)!(n-i)!}\left[\frac{\partial}{\partial t}\int [1-e^{-x_i}]^{i-1}[e^{-x_i}]^{(n-i+1-t)x_i}\right]_{t=0}\\
& = \frac{n!}{(i-1)!(n-i)!}\left[ \frac{(i-1)!(n-i)!}{n!} \sum_{k=1}^i \frac{1}{n-k-t+1} \right]_{t=0}\\
& = \sum_{k=1}^i \frac{1}{n-t+1}
\end{aligned}
\end{equation*}
If I recall correctly in my messing around I found this method works well for the variance too, but I ran into difficulties in attempting to make it work for the covariance, e.g. $\frac{\partial^2}{\partial t_1\partial t_2}e^{x_{i:n}t_1 + x_{j:n}t_2}f(x_{i:n},x_{j:n})$, but to no avail. There might be some esoteric covariance formula for ordered statistics which makes it work though.
Substitution approach
Following a paper entitled ORDER STATISTICS OF UNIFORM, LOGISTIC AND EXPONENTIAL DISTRIBUTIONS by Okoyo Collins and Omondi (see page 100-102) an alternative is to make a clever substitution;
\begin{equation*}
\begin{aligned}[b]
Z_i = (n-i+1)(X_i - X_{i-1}) \qquad \longrightarrow \qquad X_i = \frac{Z_i}{n-i+1} + X_{i-1}
\end{aligned}
\end{equation*}
which can be used to show that,
\begin{equation*}
\begin{aligned}[b]
X_i \sim \frac{Z_1}{n} + \frac{Z_2}{n-1} +...+ \frac{Z_i}{n-i+1}
\end{aligned}
\end{equation*}
The Jacobian of the transformation turns out to be $n!$ (see pg 101 of referenced paper). Also, we conveniently have,
\begin{equation*}
\begin{aligned}[b]
\sum_{i=1}^n x_i = \sum_{i=1}^n z_i
\end{aligned}
\end{equation*}
(Convince yourself of this). The joint pdf then transforms as,
\begin{equation*}
\begin{aligned}[b]
f_{X_1,X_2,...,X_n} = n!e^{-\sum_{i=1}^n x_i } \quad \longrightarrow \quad e^{-\sum_{i=1}^n z_i }
\end{aligned}
\end{equation*}
Writing the subscripts out in full ordered notation, we now have,
\begin{equation*}
\begin{aligned}[b]
E[X_{i:n}] &= E\left[\frac{Z_{1:n}}{n} + \frac{Z_{2:n}}{n-1} +...+ \frac{Z_{i:n}}{n-i+1}\right]
&= \sum_{k=1}^i \frac{1}{n-t+1}
\end{aligned}
\end{equation*}
Because $Z_{i:n} \sim EXP(1)$ as well (?). This warrants additional justification, but I've taken it as far as needed for my purposes which was applied to a different problem.
