This answer is only partial, but I hope it helps.
It depends on whether you are referring to a strictly (strong) stationary process or a weakly stationary process.
In the case of a strictly stationary process, the probabilistic behavior of a series will be identical to that of that series at any number of lags.
However, since this is a very strong assumption, the word "stationary" is often used to refer to weak stationarity. In this case, the expectation must be constant and not dependent on time t. The autocovariance must depend solely on the lag, j. Though this provides useful properties, it does not guarantee that the series' expectations and variances will be the same.