Operations on Random Variables vs Distributions vs Random Samples

Question

What is the difference between i) random variables, ii) distributions of random variables, and iii) random samples?

While trying to figure out how to average random samples from various random variables I learned that the operations summation for random variables and distributions of random variables are different. I would like to make sure I have these terms straight and I would like to make sure the operations are valid for each term.

1. Random variables: I think of these, non-formally, as functions. They are functions that can generate infinitely long vectors of numbers. In R, I think of random variables as rnorm or rexp. Random variables are functions that spit out numbers as defined by their distribution.

2. Distributions of Random Variables: In text books these are functions (PDFs). In R these would be dnorm and dexp. An example realization of a continuous distribution on a grid of evenly spaced values would be a two column set of data where one column represents an input value and the other column represents an output frequency of the associated value.

3. Random Samples: This is a vector of values output from a random variable. For example, if rnorm(n=10, 5, 1) is the random variable then the resulting vector of data is a random sample. In other words, the 10 random numbers generated from the function are the random samples. This is in contrast to random variables and distributions of random variables which are represented as functions. I do not believe that random samples must be generated from a named probability distribution. They can just be a random heap of numbers.

Operations

Random Variables

---

For Random Variables there is a wikipedia page Algebra Of Random Variables.

Considering two random variables X and Y, the following algebraic operations are possible:

• Addition: $Z=X+Y=Y+X$
• Subtraction: $Z=X-Y=-Y+X$
• Multiplication: $Z=XY=YX$
• Division (mathematics)|Division: $Z=X / Y=X \cdot (1/Y)=(1/Y) \cdot X$
• Exponentiation: $Z=X^Y=e^{Y\ln(X)}$

This notation does not seem useful. This is because I know that to sum two normals I would not just add realizations like the notations seem to indicate.

For example, to obtain a random variable $C = \frac{(A+B)}{2}$ where the random variable $A$ is :$f_A(x) = \int_{y=\infty}^{x} N(y\mid 10, 5) \, dy$ and the random variable B is:$ f_B(x) = \int_{y=\infty}^{x} N(y\mid 20, 5) \, dy$. The notation does not let the user know how to treat the parameters $\mu$ or $\sigma$ in $N(y \mid \mu, \sigma)$. It turns out that the $\mu_C$ is the average of $\mu_A$ and $\mu_C$ while the $\sigma_{C}$ is $\sigma_A$ divided by $\sqrt{2}$. The resulting random variable $C$ is $f_C(x) = \int_{y=\infty}^{x} N(y\mid 15,\frac{5}{\sqrt{2}}) \, dy$. I would not say the operations on the wikipedia page were useful to help determine the resulting random variable $C$.

So it seems that the proper, technical, way to interact with random variables is to interact with them on the distribution level as shown the in next section.

Distributions of Random Variables (PDFs or PMFs)

---

Given that there exists some density it is possible to use Convolutions to sum the random variables. There is a list of convolutions of probability distributions.

From Why is the sum of two random variables a convolution? the accepted answer states that how one sums random variables depends on how the random variables are represented.

• In terms of probability mass functions (pmf) or probability density functions (pdf), it is the operation of convolution.

• In terms of moment generating functions (mgf), it is the (elementwise) product.

• In terms of (cumulative) distribution functions (cdf), it is an operation closely related to the convolution. (See the references.)

• In terms of characteristic functions (cf) it is the (elementwise) product.

• In terms of cumulant generating functions (cgf) it is the sum.

The first two of these are special insofar as the box might not have a pmf, pdf, or mgf, but it always has a cdf, cf, and cgf.

Because it is rare for me to have a need to operate on named random variables I do not tend to have a need for these operations.

What I do have a need to interact with often is random samples or realizations of random variables.

Random Samples

---

I do not know how to perform operations such as summation, averaging, etc. on random samples.

Example 1

Consider random sample $\textbf{a}$ and $\textbf{b}$. The data points in these random samples are $a = [1,10,100]$ and $b = [100, 10, 1]$ there are a number of ways to sum.

The way np.sum() treats the problem is that it will match the index in the summation. a[0] + b[0] then a[1] + b[1] then a[2] + b[2],$a_i + b_i$, but this misses on the tails for example a[2] + b[0] = 200. The cartesian product would capture the full width of the final distribution.

I thought of a couple ways to go about this:

1. The Cartesian product
2. The bootstrap approximation

The Cartesian product gets computationally harder for larger $n$ it also can't accept different mapping structures like covariance or past errors.

As far as I know there is not a lot of guidance on working with random samples.

The Cartesian product and the bootstrap approximation was something I thought of on the fly. Does something more rigorous exist? I am basically looking for a reference for operations on random samples.

Summary of my questions

1. Am I grasping the terms i) random variables, ii) distributions of random variables, and iii) random samples properly?
2. Am I grasping operations on i) random variables, ii) distributions of random variables properly?
3. How can I perform operations such as adding, averaging, etc. on random samples? To the best of my knowledge the Cartesian product and bootstrap approximations may be possible ways to go about summing and averaging many independent random samples.

Answer 1

Random variables and their distributions cannot be considered separately. One defines another. To find the distribution of the function of random variables (e.g. algebraic operations) you require convolutions, as you described in your second section on the distributions.

How are the algebraic operations on random variables themselves useful? This is explained in the very first sentence of the Wikipedia article

The algebra of random variables in statistics, provides rules for the symbolic manipulation of random variables, while avoiding delving too deeply into the mathematically sophisticated ideas of probability theory. [...]

So you can do the algebraic operations on a symbolic level without bothering about the "implementation details" while having guarantees that it works as you would expect.

As for random samples, what you are describing is not correct. To find the distribution of an operation on random variables, e.g. sum, you would need to consider their joint distribution. If you used a cartesian product, you would assume that each combination of the values has equal probability. Bootstrap approximation of the distribution using the three samples you mentioned would just give you probabilities for the values $\Pr(a + b = 101) = 2/3$ and $\Pr(a + b = 20) = 1/3$, that is not the "full width" of the distribution because you would just be randomly sampling the $(a_i, b_i)$ pairs. If you meant randomly permuting the combinations, again you would be assuming that the probabilities of each combination are equally likely.

Consider the simple joint distribution

$$\begin{align} \Pr(a=0, b=0) &= 0.2 \\ \Pr(a=0, b=1) &= 0.5 \\ \Pr(a=1, b=0) &= 0.3 \\ \Pr(a=1, b=1) &= 0 \\ \end{align}$$

In such a case you cannot claim that $\Pr(a+b = 2) > 0$, as using the proposed cartesian product approach or using random permutations, would.

With random samples, you cannot directly get the "full width" of the distribution. Depending on the problem, you could approximate the distribution of the function of the samples using some statistical model. Your example considers only three samples with two unique values for the sum, so there is not much room to build any reasonable statistical model using such data.

If you had enough data, their empirical distribution approximates the true distribution. In such a case, the distribution of the empirical results of evaluating some function over the samples would approximate the distribution of the function of the random variable. But you cannot easily go out-of-sample as you suggested.

Answer 2

In questions like these it is best to keep everything as elementary as possible. We will assume that everything is finite, if you are not sure what I mean by that, then the examples will illustrate.

---

Let us first begin by defining a "random experiment". This is any experiment whose outcome cannot be determined until it is done. For example, if you have a coin and you measure how long it takes for the coin to fall and reach the floor from 1 meter height, then the answer will (roughly) by always the same no matter how many times you do it. There are deterministic laws which tell us exactly what will happen and we can predict ahead of time of what will happen. An experiment is "random" if its outcome cannot be predicted in any consistent manner. Going back to the coin, if instead we wish to determine whether the coin lands head/tails, then that cannot be predicted at all. Hence we call it "random".

We define the "sample space", commonly denoted as $\Omega$, to be the set of all possible outcomes that can occur under some random experiment. For example, if we have two coins, a quarter and a dime, then we have 4 outcomes that can happen, $hH$, $hT$, $tH$, $tT$, where the little letter is for the dime and the big letter is for the quarter, i.e. $hT$ represents that the dime lands on heads and the quarter lands on tails. Therefore, $\Omega = \{ hH, hT, tH, tT\}$. This is our "sample space" of the random experiment, anything that can happen is contained in that universe of possibilities.

---

Now let us define a ``random variable''. A random variable, let us denote it as $X$, is a mapping which assigns a numerical quantity for each sample point in $\Omega$. If $\omega\in \Omega$ then $X(\omega)$ is the assigned value to that sample point $\omega$. For example, let us go back to our dime/quarter sample space. A simple example of a random variable $X$ on that sample space is simply the monetary value (in cents) of that outcome of heads only. So, for instance, $X(hT) = 10$ and $X(hH) = 35$, ect.

The purpose of a "random variable" is to quantify the sample space. As we can see in our example, $\Omega$ is not a set of numbers, it is just a set of symbolics, a random variable simply assigns numerical values to each of those.

---

Now observe something very important. If you were to do the random experiment, over and over and over again, many many times, the value of $X$ (the random variable) will be ``random''. It will be unpredictable. This is what we call "random samples" from $X$. It is a list of random outcomes that the random variable $X$ produced. For example, here is a list of random samples: $$ 0, 10, 25, 0, 25, 25, 35, 10 $$ What this means is that on the first random experiment the value of $X$ was 0 cents. On the second time you did the experiment it was 10 cents, ect.

---

Finally let us get to what is known as the "distribution" of $X$. This is were probability comes in. It would be inappropriate to ask "what is $X$ equal to?", since that cannot be predicted. Instead, we can ask, "what is the probability that $X$ is equal to $0$ cents?". Then we see that this outcome, of zero cents, only happens at the sample point $tT$, so $P(X = 0) = \tfrac{1}{4}$.

The way we find the distribution is that we first determine what is the set of all possible outcomes of $X$ (sorta like, a sample space for $X$ values) and we see that this is equal to $\{0, 10, 25, 35\}$. We call that the "support" of $X$. We use the notation, $\text{supp}(X) = \{0,10,25,35\}$. Next we determine the probability that our random variable $X$ will be equal to any one of those four, in our simple example, $P(X = n) = \tfrac{1}{4}$ if $n \in \text{supp}(X)$ and $P(X=n) = 0$ if $n\not \in \text{supp}(X)$. The "distribution" of $X$ is a function (not a random variable!), let us denote it as $f_X(\cdot)$, which is defined as, $$ f_X(n) = \left\{ \begin{array}{ccc} 0.25 & \text{ if } & n = 0,10,25,35 \\ 0 & \text{ if } & \text{otherwise} \end{array} \right. $$ The purpose of the distribution $f_X(\cdot)$ is that is computes the probabilities associated with the behavior of the random variable $X$.