Supremum of parameterized random variables over compact set Suppose that we have a parameterized real-valued discrete stochastic process  $x(t) :=\{x_k(t)\}_{k=1}^\infty$, such that $t$ assumes values in a compact set $T\subset \mathbb{R}^d$ for some finite positive integer $d$ (say = 1 for simplicity). Assume that for every $t\in T$, the process $x(t)$ is a second order process; i.e it has finite mean and variance functions. 
My question is the following: What are the properties of the random variable defined as $\sup_{t\in T} x_k(t)$ (I am assuming that it is well-defined(?)) and the process defined as $x := \{\sup_{t\in T} x_k(t)\}_{k=1}^\infty$? Since we say that $x(t)$ is a second order process for all $t \in T$, I am wondering if this carry on to the random variables defined by taking the $\sup$, or something may go wrong?
To simplify the situation as much as possible and be able to understand the meaning of the defined process, let $x(t)$ be a nice function in $t$, say continuous over $T$. Is $x$ a second order process?  i.e., do the random variables $x_k := \sup_{t\in T} x_k(t)$ have finite mean and variance functions?
 A: Something can go wrong.
First, let's simplify a bit: the index $k$ is superfluous, so let's either drop it or focus on a fixed value of $k$ for the duration.  Since all that really matters is that $T$ is compact and all paths $t\to x(t)$ are continuous, we might as well study the unit interval $T=[0,1]$ with its usual (Euclidean) topology.  This guarantees the random variable $Z=\sup_{t\in T} x(t)$ is finite and is attained by at least one $t\in T.$
The question concerns whether the finite variance of every random variable $x(t)$ guarantees the finite variance of $Z.$  How could it not?  A little reflection suggests a problem might occur if any small subsets $S\subset T$ could occasionally exhibit huge values of $x,$ but would do so extremely rarely, thereby keeping the variances of the $x(t)$ all finite.  This suggests investigating the contrapositive: what would an infinite variance of $Z$ imply about the variances of the $x(t)$?  Must they then be infinite also?
The answer is no, as I will show by constructing a counterexample. It begins by defining nice functions--say, continuous ones--whose values are universally bounded except for spikes on small subsets $S \subset T.$  We might as well make this universal bound on $T\setminus S$ equal to $0.$  Consider, then, any small interval that extends some distance $\sigma$ to either side of a number $\mu;$ namely, $S = [\mu-\sigma,\mu+\sigma].$  Let $\phi:\mathbb{R}\to[0,1]$ be a continuous decreasing function for which $\phi(0)=1$ and $\phi(1)=0.$  For instance, $\phi$ could be the survival function of any continuous random variable supported on $[0,1].$

By translating and rescaling the argument $t,$ applying $\phi$ to $|t|,$ and rescaling its result by some (positive) value $Z,$ we can create a "nice" spike supported in the interval $S$ rising from $0$ to $Z.$  This spike attains the value $Z$ inside $S$ and outside of $S$ is identically zero.

Since we're searching for a counterexample--that is, a process where the random variable $Z$ has infinite variance--let's begin with $Z$ itself: suppose it has a positively supported distribution $F(z)=\Pr(Z\le z)$ with infinite variance (which implies an infinite mean).  Set $n = n(Z)$ to be the smallest integer greater than or equal to $Z$.  Take a whole number $p\ge 1$ (we'll determine its value later) and partition the interval $T$ into $N=N_p(Z)=n(Z)^p$ smaller intervals
$$I_{k;N} = \left[\frac{k}{N}, \frac{k+1}{N}\right)$$
for $k=0, 1, 2, \ldots, N-1.$  (The final interval for $k=N-1$ needs to include its right endpoint at $1.$)
Now pick $k$ randomly and uniformly from its $N$ possible values, independently of $Z,$ and define the function $x$ to be the spike of height $Z$ supported on $I_{k;N}.$  Notice that any specified $t\in T$ has a chance of exactly $1/N(z)$ of being within one of these intervals.
To illustrate this construction, I drew four random variables $Y_i$ from a Cauchy distribution and set $Z_i=1+|Y_i|.$  The $Z_i$ have infinite mean, but these four particular realizations happened to be $z_i = (1.78, 2.53, 1.01, 1.18).$  Their greatest integers are $n=(2,3,2,2).$  I selected $p=3$ for this illustration, whence $N=n^3 = (8,27,8,8).$  The randomly selected values of $k$ were $k = (5,20,3,6),$ yielding these four functions in order:

The spikes peak at the values $(5+1/2)/8, (20+1/2)/27, (3+1/2)/8,$ and $(6+1/2)/8,$ respectively.
I have described a stochastic process $x(t)$ having all the desired properties: by construction, every realization is a continuous real-valued function of $T,$ these realizations are governed by a definite probability distribution, and the distribution of $\sup_{t\in T}x(t)$ is $F.$  This process is determined by the distribution $F$ and an integer $p \ge 1.$
We need to consider the variance of the random variable associated with any point $t\in T;$ namely, $\operatorname{Var}(x(t)).$  To do this, let's estimate the survival function of $x(t),$ which I will call $S_p$ (to remind us that the answer will depend on $p$).
For any $z\gt 0$ and $t\in T,$ the only way $x(t)$ can possibly exceed $z$ is for $t$ to be near a spike taller than $z.$  This means $Z$ exceeds $z$ and $t$ lies in the random interval $I_{k;N(Z)}.$  (Even then it's possible (when $t$ is near the endpoints of that interval) for $x(t)$ still to be less than $z.$)  This justifies the first inequality in the following approximation:
$$\eqalign{
S_p(z)&=\Pr(x(t) \gt z) \\
&\le \Pr(t \in I_{k;N_p(Z)}\text{ and } Z \gt z) \\
&= \Pr(t \in I_{k;N_p(Z)} \mid Z \gt z) \Pr(Z \gt z) \\
&\le \frac{1}{N_p(z)}\Pr(Z\gt z)  \\
&\le \frac{1-F(z)}{z^p}.
}$$
The second inequality follows because $N_p$ is an increasing function and the last inequality is a consequence of $N_p(z) \ge z^p.$
Since for any distribution, $1-F(z)\le 1,$ we obtain
$$S_p(z) \le z^{-p}.$$
Obviously $S_p(z)\le 1,$ too.
Since the variance is finite if and only if the (raw) second moment is finite, and an integration by parts gives
$$\mathbb{E}(x(t)^2) = 2\int_0^\infty x S_p(x) \mathrm{d}x \le 2\left(\int_0^1 x (1) \mathrm{d}x + \int_1^\infty x(x^{-p})\mathrm{d}x\right) = \frac{p}{p-2}\lt\infty$$
for all $p \gt 2,$ it follows that 

For every $t\in T,$ $x(t)$ has finite variance (and therefore finite mean, too) provided $p\gt 2.$  Nevertheless, the distribution of $Z = \sup_{t\in T} x(t)$ is arbitrary and therefore can have infinite variance and even infinite mean.


To show how concrete this counterexample is, and to help the reader study it, here is R code to simulate the process $x(t).$  Of course it cannot create complete functions $x,$ but given a specified set of arguments $t_1, t_2, \ldots, t_n,$ it will compute the values $x(t_1), \ldots, x(t_n)$ for n.sim independent realizations of the process.  To illustrate how the output can be analyzed, at the end it prints the average simulated value of $Z$ (the supremum process) and the average, by $t_i,$ of the simulated values $x(t_i).$  The mean of the simulated values of $Z$ will vary widely among simulations because $Z$ has infinite mean, but a typical average is around 10, while typical averages of the $x(t_i)$ are stable around 0.25.
phi <- function(t, z, mu, sigma) {
  x <- abs((t - mu) / sigma)
  ifelse(x < 1, 1-x^2*(3-2*x), 0) * z
}
#
# Create realizations of the process `x` and track them at specified points `t`.
#
n.sim <- 1e4
p <- 3
n <- 19                        # Number of points to track
# t <- signif((1:n-1/2)/n, 1)  # Values of 't' to track
t <- signif(sort(runif(n)), 2)
#
# Simulate the random variables (Z, k).
#
z <- 1 / runif(n.sim) # A positive random variable with infinite mean and variance
N <- ceiling(z^p)     # (In the text, ceiling(z)^p was used.)
k <- floor(runif(length(N), 0, N))
#
# Compute the x(t) based on the random variables (Z, k).
#
sigma <- 1/N
mu <- (k + 1/2)/N
x <- sapply(t, function(t) phi(t, z, mu, sigma))
#
# Create a data frame for plotting and further analysis.
#
X <- rbind(data.frame(x=c(x),
                      t=rep(t, each=n.sim),
                      iteration=rep(1:n.sim, length(t))),
           data.frame(x=z,
                      t=NA,
                      iteration=1:n.sim)
)
colMeans(x)
mean(z)    # Will be much larger than any of the x means
sd(z)      # Will be huge

