Supremum of parameterized random variables over compact set

Suppose that we have a parameterized real-valued discrete stochastic process $$x(t) :=\{x_k(t)\}_{k=1}^\infty$$, such that $$t$$ assumes values in a compact set $$T\subset \mathbb{R}^d$$ for some finite positive integer $$d$$ (say = 1 for simplicity). Assume that for every $$t\in T$$, the process $$x(t)$$ is a second order process; i.e it has finite mean and variance functions.

My question is the following: What are the properties of the random variable defined as $$\sup_{t\in T} x_k(t)$$ (I am assuming that it is well-defined(?)) and the process defined as $$x := \{\sup_{t\in T} x_k(t)\}_{k=1}^\infty$$? Since we say that $$x(t)$$ is a second order process for all $$t \in T$$, I am wondering if this carry on to the random variables defined by taking the $$\sup$$, or something may go wrong?

To simplify the situation as much as possible and be able to understand the meaning of the defined process, let $$x(t)$$ be a nice function in $$t$$, say continuous over $$T$$. Is $$x$$ a second order process? i.e., do the random variables $$x_k := \sup_{t\in T} x_k(t)$$ have finite mean and variance functions?

Something can go wrong.

First, let's simplify a bit: the index $$k$$ is superfluous, so let's either drop it or focus on a fixed value of $$k$$ for the duration. Since all that really matters is that $$T$$ is compact and all paths $$t\to x(t)$$ are continuous, we might as well study the unit interval $$T=[0,1]$$ with its usual (Euclidean) topology. This guarantees the random variable $$Z=\sup_{t\in T} x(t)$$ is finite and is attained by at least one $$t\in T.$$

The question concerns whether the finite variance of every random variable $$x(t)$$ guarantees the finite variance of $$Z.$$ How could it not? A little reflection suggests a problem might occur if any small subsets $$S\subset T$$ could occasionally exhibit huge values of $$x,$$ but would do so extremely rarely, thereby keeping the variances of the $$x(t)$$ all finite. This suggests investigating the contrapositive: what would an infinite variance of $$Z$$ imply about the variances of the $$x(t)$$? Must they then be infinite also?

The answer is no, as I will show by constructing a counterexample. It begins by defining nice functions--say, continuous ones--whose values are universally bounded except for spikes on small subsets $$S \subset T.$$ We might as well make this universal bound on $$T\setminus S$$ equal to $$0.$$ Consider, then, any small interval that extends some distance $$\sigma$$ to either side of a number $$\mu;$$ namely, $$S = [\mu-\sigma,\mu+\sigma].$$ Let $$\phi:\mathbb{R}\to[0,1]$$ be a continuous decreasing function for which $$\phi(0)=1$$ and $$\phi(1)=0.$$ For instance, $$\phi$$ could be the survival function of any continuous random variable supported on $$[0,1].$$

By translating and rescaling the argument $$t,$$ applying $$\phi$$ to $$|t|,$$ and rescaling its result by some (positive) value $$Z,$$ we can create a "nice" spike supported in the interval $$S$$ rising from $$0$$ to $$Z.$$ This spike attains the value $$Z$$ inside $$S$$ and outside of $$S$$ is identically zero.

Since we're searching for a counterexample--that is, a process where the random variable $$Z$$ has infinite variance--let's begin with $$Z$$ itself: suppose it has a positively supported distribution $$F(z)=\Pr(Z\le z)$$ with infinite variance (which implies an infinite mean). Set $$n = n(Z)$$ to be the smallest integer greater than or equal to $$Z$$. Take a whole number $$p\ge 1$$ (we'll determine its value later) and partition the interval $$T$$ into $$N=N_p(Z)=n(Z)^p$$ smaller intervals

$$I_{k;N} = \left[\frac{k}{N}, \frac{k+1}{N}\right)$$

for $$k=0, 1, 2, \ldots, N-1.$$ (The final interval for $$k=N-1$$ needs to include its right endpoint at $$1.$$)

Now pick $$k$$ randomly and uniformly from its $$N$$ possible values, independently of $$Z,$$ and define the function $$x$$ to be the spike of height $$Z$$ supported on $$I_{k;N}.$$ Notice that any specified $$t\in T$$ has a chance of exactly $$1/N(z)$$ of being within one of these intervals.

To illustrate this construction, I drew four random variables $$Y_i$$ from a Cauchy distribution and set $$Z_i=1+|Y_i|.$$ The $$Z_i$$ have infinite mean, but these four particular realizations happened to be $$z_i = (1.78, 2.53, 1.01, 1.18).$$ Their greatest integers are $$n=(2,3,2,2).$$ I selected $$p=3$$ for this illustration, whence $$N=n^3 = (8,27,8,8).$$ The randomly selected values of $$k$$ were $$k = (5,20,3,6),$$ yielding these four functions in order:

The spikes peak at the values $$(5+1/2)/8, (20+1/2)/27, (3+1/2)/8,$$ and $$(6+1/2)/8,$$ respectively.

I have described a stochastic process $$x(t)$$ having all the desired properties: by construction, every realization is a continuous real-valued function of $$T,$$ these realizations are governed by a definite probability distribution, and the distribution of $$\sup_{t\in T}x(t)$$ is $$F.$$ This process is determined by the distribution $$F$$ and an integer $$p \ge 1.$$

We need to consider the variance of the random variable associated with any point $$t\in T;$$ namely, $$\operatorname{Var}(x(t)).$$ To do this, let's estimate the survival function of $$x(t),$$ which I will call $$S_p$$ (to remind us that the answer will depend on $$p$$).

For any $$z\gt 0$$ and $$t\in T,$$ the only way $$x(t)$$ can possibly exceed $$z$$ is for $$t$$ to be near a spike taller than $$z.$$ This means $$Z$$ exceeds $$z$$ and $$t$$ lies in the random interval $$I_{k;N(Z)}.$$ (Even then it's possible (when $$t$$ is near the endpoints of that interval) for $$x(t)$$ still to be less than $$z.$$) This justifies the first inequality in the following approximation:

\eqalign{ S_p(z)&=\Pr(x(t) \gt z) \\ &\le \Pr(t \in I_{k;N_p(Z)}\text{ and } Z \gt z) \\ &= \Pr(t \in I_{k;N_p(Z)} \mid Z \gt z) \Pr(Z \gt z) \\ &\le \frac{1}{N_p(z)}\Pr(Z\gt z) \\ &\le \frac{1-F(z)}{z^p}. }

The second inequality follows because $$N_p$$ is an increasing function and the last inequality is a consequence of $$N_p(z) \ge z^p.$$

Since for any distribution, $$1-F(z)\le 1,$$ we obtain

$$S_p(z) \le z^{-p}.$$

Obviously $$S_p(z)\le 1,$$ too.

Since the variance is finite if and only if the (raw) second moment is finite, and an integration by parts gives

$$\mathbb{E}(x(t)^2) = 2\int_0^\infty x S_p(x) \mathrm{d}x \le 2\left(\int_0^1 x (1) \mathrm{d}x + \int_1^\infty x(x^{-p})\mathrm{d}x\right) = \frac{p}{p-2}\lt\infty$$

for all $$p \gt 2,$$ it follows that

For every $$t\in T,$$ $$x(t)$$ has finite variance (and therefore finite mean, too) provided $$p\gt 2.$$ Nevertheless, the distribution of $$Z = \sup_{t\in T} x(t)$$ is arbitrary and therefore can have infinite variance and even infinite mean.

To show how concrete this counterexample is, and to help the reader study it, here is R code to simulate the process $$x(t).$$ Of course it cannot create complete functions $$x,$$ but given a specified set of arguments $$t_1, t_2, \ldots, t_n,$$ it will compute the values $$x(t_1), \ldots, x(t_n)$$ for n.sim independent realizations of the process. To illustrate how the output can be analyzed, at the end it prints the average simulated value of $$Z$$ (the supremum process) and the average, by $$t_i,$$ of the simulated values $$x(t_i).$$ The mean of the simulated values of $$Z$$ will vary widely among simulations because $$Z$$ has infinite mean, but a typical average is around 10, while typical averages of the $$x(t_i)$$ are stable around 0.25.

phi <- function(t, z, mu, sigma) {
x <- abs((t - mu) / sigma)
ifelse(x < 1, 1-x^2*(3-2*x), 0) * z
}
#
# Create realizations of the process x and track them at specified points t.
#
n.sim <- 1e4
p <- 3
n <- 19                        # Number of points to track
# t <- signif((1:n-1/2)/n, 1)  # Values of 't' to track
t <- signif(sort(runif(n)), 2)
#
# Simulate the random variables (Z, k).
#
z <- 1 / runif(n.sim) # A positive random variable with infinite mean and variance
N <- ceiling(z^p)     # (In the text, ceiling(z)^p was used.)
k <- floor(runif(length(N), 0, N))
#
# Compute the x(t) based on the random variables (Z, k).
#
sigma <- 1/N
mu <- (k + 1/2)/N
x <- sapply(t, function(t) phi(t, z, mu, sigma))
#
# Create a data frame for plotting and further analysis.
#
X <- rbind(data.frame(x=c(x),
t=rep(t, each=n.sim),
iteration=rep(1:n.sim, length(t))),
data.frame(x=z,
t=NA,
iteration=1:n.sim)
)
colMeans(x)
mean(z)    # Will be much larger than any of the x means
sd(z)      # Will be huge

• Thanks a lot for the answer! I think I need and will read it couple of times to follow the arguments. But I already have few questions: Are you looking at a continuous stochastic process $\{x_k(t)\}_{t\in T}$ which is indexed by $t$ (and $k$ is fixed)? I'm confused a little because I thought that (and you also stated that), for every $k$, $Z_k := \sup_{t\in T} x_k(t)$ is attained by one $t$. Doesn't this mean that $\exists t^\star_k\in T : Z_k = x_k(t^\star)$? But then $Z_k$ has to be second order because $x_k(t)$ is second order for every $k$ and all $t\in T$, $Z_k$ . What am I missing? May 22, 2019 at 22:27
• How do you conclude "But then..."?
– whuber
May 22, 2019 at 22:28
• "... because $x_k(t)$ is second order for every $k$ and all $t\in T$..." by assumption and the fact that the continuity in $t$ over the compact $T$ implies the equality $Z_k = x_k(t^\star_k)$ for some $t_k^\star \in T$, which means that these two random variables have the same distribution. May 22, 2019 at 22:33
• I'm afraid it doesn't imply that: that's basically the whole point. The problem is that there aren't just two random variables involved here, because $t$ itself depends on $Z$ and therefore is random. Maybe a different kind of example will help make this point. Flip a fair coin and set $x(t)=1$ for all $t$ if the coin is heads and otherwise set $x(t)=0.$ The distribution of every $x(t)$ is Bernoulli$(1/2),$ but $Z$ is identically $1,$ which obviously does not have a Bernoulli$(1/2)$ distribution. This is a counterexample to your claim that $Z$ must have the same distribution as some $x(t).$
– whuber
May 22, 2019 at 22:38
• Let me write the random variables as follows: $x(t,k, \omega)$ where $\omega \in \Omega$ (the sample space). Do you mean that one should write $t_k^\star(\omega) = \arg\max_{t\in T} x(t,k, \omega)$ ? Also in this last example you gave, what is $T$? Just need to see that $x$ is continuous in $t$ over $T$. Many thanks! May 22, 2019 at 23:00