6
$\begingroup$

BACKGROUND: This is probably an extremely simple question to answer, but it's one of the downsides of being self-taught to get ensnarled in unexpected difficulties with basic stuff. This question stems from the last sentence in this post and the first sentence in the answer.

QUESTION: The definition of a random vector is not that difficult - a list of random variables. Or, using the nice explanation by @whuber on the same post, we can tell that a vector is random by contemplating "what changes would occur if the data-collection process were replicated."

Part of the problem is understanding why the coefficients of the model matrix are treated as constants in regression. even if they are random variables (columns of a dataset).

And probably this is connected somehow to my difficulties seeing why non-random vectors have zero covariance.

In the end, given two numeric vectors of equal length, it seems like it should always be possible to get their means and perform the operation: $(X_i -\bar X)\,(Y_i -\bar Y)$ an eventually get the mean of the resulting multiplications, seemingly following the equation $cov(\mathbf{X,Y}) =\mathbb{E}\left( (\mathbf{X}-\mathbb{E}(\mathbf{X}))(\mathbf{Y}-\mathbb{E}(\mathbf{Y}))\right)$.

I am sure there are many misunderstandings in these statements, but this is the question: Why is the covariance of non-random vectors always zero when it seems as though given any pair of numeric vectors of equal length, we can get their covariance? It may have to due with the difference between deriving the covariance of a probability density function versus the covariance between samples.

In this sense I can calculate the covariance between two deterministically calculated vectors without any noise, corresponding to power functions of $x$. The resultant vectors are not random, yet the correlation values still make some sense as a measure of proximity:

par(mfrow = c(1,2))

x <- seq(1, 10, by = 0.1)
z <- seq(-3.1, -0.1, by = 0.1)
o <- x^-3.1
plot(o ~ x, col ='blue4', pch = 19, cex=0.1,
     ylab = "x to i power")

for (i in seq(z)){
    b <- x^(z[i])
    points(b  ~  x, pch = 20, cex = 0.1, col = i)
}


COR <- 0
for (i in seq(z)){
    b <- x^z[i]
    COR[i] <- cov(b, o)
}
plot(COR ~ z, type="l", lwd=2, col='blue4', 
     xlab="Exponent", ylab = 'COVARIANCE')

enter image description here

Is it purely a definitional type of difference - ... by definition covariance only applies to random variables...?

I can see that a very straightforward way to prove this is to think that in a constant vector $\mathbb{E}(\mathbf{X}) = \mathbf{X}$, and hence, $\mathbf{X}-\mathbb{E}(\mathbf{X})=0$, so perhaps the question really boils down to seeing what constitutes a random vector on a data set.


INITIAL SELF-ANSWER ATTEMPT:

[I am posting a very tentative answer in the hope of being corrected, and because I don't think that the opposite of a random vector is a vector with identical elements in every position as an example in the answers given seem to imply.]

Perhaps the answer is as simple as to say that it is not a computational difference. Random and nonrandom vectors are indistinguishable unless we know how they should be interpreted: a vector is random if the assumption is that it represents a sample from a population.

Random vector are a collection of random variables associated with the same events.

For instance in the fictional dataset:

     Subject    Weight   BP    Glucose
1      AG       194      85      99
2      ST       185      86      108
3      PS       180      81      102
4      SS       167      87      100

a 2-dimensional random vector can be defined as:

$\small \color{red}{\mathbf{V}} = \begin{bmatrix}\text{Weight}\\\text{BP}\end{bmatrix}$

and a sample from this random vector is precisely what the data frame contains - in this case 4 observations:

$\small \color{red}{\text{sample of}\,\mathbf{V}}=\color{blue}{\mathbf{V}}=\begin{bmatrix}194&185&180&167\\ 85&86&81&87\end{bmatrix}$

We can estimate the mean of the population as $\small\color{blue}{\mathbf{\bar V}} = \small\frac{1}{n}\displaystyle \sum_{i=1}^n\,\begin{bmatrix}\text{W}_i\\\text{BP}_i\end{bmatrix} =\begin{bmatrix}181.5\\84.75\end{bmatrix}$

And use this sample mean to center $\color{ blue}{\mathbf V}$ by subtracting column wise $\color{blue}{\mathbf{\bar V}}$ from $\color{blue}{\mathbf V}$. This resultant center matrix could be called $\color{blue}{\mathbf V_c}$.

In this way the covariance can be expressed as

$\small \operatorname{cov}(\color{blue}{\mathbf{V}})=\frac{1}{n-1}\small\color{blue}{\mathbf{V_c} \mathbf{V_c'}}=\frac{1}{3}\begin{bmatrix}\operatorname{var}(\text{W})&\operatorname{cov}(\text{W,BP})\\\operatorname{cov}(\text{W,BP})&\operatorname{var}(\text{BP})\end{bmatrix}=\Tiny\begin{bmatrix}127&-6.5\\-6.5&6.9\end{bmatrix}$.

These operations could naturally be performed on vectors that did not share the link of being observations on the same subjects, as in this case, and subject to noise as random variables. Or if they were deterministically calculated through algebraic equations as in the example in the OP.

However, it seems as though there is simply no real need to extrapolate the concept of covariance to deterministic vectors, even beyond the obvious violation of the term "co - variance" when there is no variance in a nonrandom vector. For instance, in the case illustrated on the plot, the difference between the algebraic functions generating the vectors, would supersede any need for the "covariance" between the vectors. And I supposed the arithmetic subtraction of two otherwise unspecified deterministic vectors would do the same.

I found the discussion in this forum of help, and to avoid broken links I will paste a couple of answers:

"Let's see, "mean", "variance" and "covariance" can be defined for samples of random variables or for the random variables themselves. When they are defined for the random variables themselves they are computed from the distribution functions for the random variables by integrations ( counting summation as a type of integration). So these procedures are performed on a specific function (which is what the original post means by a "deterministic" function, I suppose.) The same sort of integrations can be applied to a function that is not a distribution function."

"Perhaps the original post concerns whether this is every done [sic] and whether the things that are computed this way are still called "mean", "variance" and "covariance"."

"I think a function that is not a distribution can have a "first moment" which is computed like a mean. It can have an "L2-norm" which is like the square root of the sum of the squares. So I think the same sorts of integrations are indeed done on functions that are not distribution functions. Whether a given field (like signal processing) uses the terminology "mean", "variance" and "covariance" for the things computed, I don't know."

Reference https://www.physicsforums.com/threads/what-is-covariance-with-both-random-and-deterministic-variables.561504/"

$\endgroup$
  • 1
    $\begingroup$ Isn't this question a duplicate of stats.stackexchange.com/questions/172498/…? How does it differ? $\endgroup$ – whuber Nov 10 '15 at 14:48
  • $\begingroup$ @whuber Any chance you could briefly address the issue, linking to the answer on your comment, and clarifying whether there is any validity to the answers I got from other users? In this way Glen's points wouldn't go to waste, and the "noise" in the answers could be dispelled. $\endgroup$ – Antoni Parellada Nov 10 '15 at 19:36
  • $\begingroup$ At this point, your answer is the only one that attempts to address the question, but I'll give it a quick try. $\endgroup$ – whuber Nov 10 '15 at 20:20
7
+250
$\begingroup$

"Covariance" is used in many distinct senses. It can be

  1. a property of a bivariate population,

  2. a property of a bivariate distribution,

  3. a property of a paired dataset, or

  4. an estimator of (1) or (2) based on a sample.

Because any finite collection of ordered pairs $((x_1,y_1), \ldots, (x_n,y_n))$ can be considered an instance of any one of these four things--a population, a distribution, a dataset, or a sample--multiple interpretations of "covariance" are possible. They are not the same. Thus, some non-mathematical information is needed in order to determine in any case what "covariance" means.


In light of this, let's revisit three statements made in the two referenced posts:

If $u,v$ are random vectors, then $\operatorname{Cov}(u,v)$ is the matrix of elements $\operatorname{Cov}(u_i,v_j).$

This is complicated, because $(u,v)$ can be viewed in two equivalent ways. The context implies $u$ and $v$ are vectors in the same $n$-dimensional real vector space and each is written $u=(u_1,u_2,\ldots,u_n)$, etc. Thus "$(u,v)$" denotes a bivariate distribution (of vectors), as in (2) above, but it can also be considered a collection of pairs $(u_1,v_1), (u_2,v_2), \ldots, (u_n,v_n)$, giving it the structure of a paired dataset, as in (3) above. However, its elements are random variables, not numbers. Regardless, these two points of view allow us to interpret "$\operatorname{Cov}$" ambiguously: would it be

$$\operatorname{Cov}(u,v) = \frac{1}{n}\left(\sum_{i=1}^n u_i v_i\right) - \left(\frac{1}{n}\sum_{i=1}^n u_i\right)\left(\frac{1}{n}\sum_{i-1}^n v_i\right),\tag{1}$$

which (as a function of the random variables $u$ and $v$) is a random variable, or would it be the matrix

$$\left(\operatorname{Cov}(u,v)\right)_{ij} = \operatorname{Cov}(u_i,v_j) = \mathbb{E}(u_i v_j) - \mathbb{E}(u_i)\mathbb{E}(v_j),\tag{2}$$

which is an $n\times n$ matrix of numbers? Only the context in which such an ambiguous expression appears can tell us which is meant, but the latter may be more common than the former.

If $u,v$ are not random vectors, then $\operatorname{Cov}(u,v)$ is the scalar $\Sigma u_i v_i$.

Maybe. This assertion understands $u$ and $v$ in the sense of a population or dataset and assumes the averages of the $u_i$ and $v_i$ in that dataset are both zero. More generally, for such a dataset, their covariance would be given by formula $(1)$ above.

Another nuance is that in many circumstances $(u,v)$ represent a sample of a bivariate population or distribution. That is, they are considered not as an ordered pair of vectors but as a dataset $(u_1,v_1), (u_2,v_2), \ldots, (u_n,v_n)$ wherein each $(u_i,v_i)$ is an independent realization of a common random variable $(U,V)$. Then, it is likely that "covariance" would refer to an estimate of $\operatorname{Cov}(U,V)$, such as

$$\operatorname{Cov}(u,v) = \frac{1}{n-1}\left(\sum_{i=1}^n u_i v_i - \frac{1}{n}\left(\sum_{i=1}^n u_i\right)\left(\sum_{i-1}^n v_i\right)\right).$$

This is the fourth sense of "covariance."

If two vectors are not random, then their covariance is zero.

This is an unusual interpretation. It must be thinking of "covariance" in the sense of formula $(2)$ above,

$$\left(\operatorname{Cov}(u,v)\right)_{ij} = \operatorname{Cov}(u_i,v_j) = 0$$

Each $u_i$ and $v_j$ is considered, in effect, a random variable that happens to be a constant.


In a regression context (where vectors, numbers, and random variables all occur together) some of these distinctions are further elaborated in the thread on variance and covariance in the context of deterministic values.

$\endgroup$
  • $\begingroup$ This is just about exactly the kind of answer I had been hoping to see on this question. $\endgroup$ – Glen_b Nov 11 '15 at 21:11
  • $\begingroup$ @Glen_b Thank you for your pledge! A minor point for W. Huber is that I find the emphatic "Maybe" difficult to connect with the otherwise lucid explanation. $\endgroup$ – Antoni Parellada Nov 11 '15 at 21:45
  • $\begingroup$ Thank you, Antoni. I have inserted a couple of lines to address that lacuna. $\endgroup$ – whuber Nov 11 '15 at 21:48
  • $\begingroup$ (+1) Perfect summary! $\endgroup$ – Xi'an Nov 29 '15 at 20:33
2
$\begingroup$

You have a misconception. Using the cov function on a pair of vectors computes the sample covariance of the two vectors. That is, it is assuming X and Y are vectors of observations of two one-dimensional random variables. That is why it doesn't give zero as an answer. What is zero (a matrix filled with zeroes, actually) is the covariance of two nonrandom numeric vectors. It would be silly to try to compute that covariance matrix though ^^

EDIT: Here's an example, silliness and all:

x <- matrix(rep(1:3,5), nrow=5, byrow=T)
y <- matrix(rep(c(4.3,1.2,999),5), nrow=5, byrow=T)
x
y
cov(x,y) # All zeroes
$\endgroup$
  • $\begingroup$ Note that in R "If x and y are matrices then the covariances (or correlations) between the columns of x and the columns of y are computed." $\endgroup$ – StatsStudent Oct 30 '15 at 4:45
  • $\begingroup$ So what would the random vectors on a dataframe, say for instance my favorite toy dataframe in r mtcars? $\endgroup$ – Antoni Parellada Oct 30 '15 at 4:59
  • 1
    $\begingroup$ I added an example. The trick is that if your random vector is actually constant, a matrix of "observations" of it just repeats the same vector over and over again. Also, I don't understand your last comment! $\endgroup$ – Felipe Gerard Oct 30 '15 at 5:04
  • $\begingroup$ What you are showing is a couple of matrices where each column is composed of identical elements. So it clearly would be all zeros by the same token that the sample covariance of two vectors composed of identical elements will be zero, since the mean is identical to all elements. $\endgroup$ – Antoni Parellada Oct 30 '15 at 5:10
  • $\begingroup$ It sounds as though there is some effort or reasoning behind determining whether a vector is random or not (for instance in the context of OLS, as in the posts I linked in the OP), so it can't be that trivial. $\endgroup$ – Antoni Parellada Oct 30 '15 at 5:12
1
$\begingroup$

Simply put a non-random vector has component elements that are constant -- they do not change by definition. Since they do not change, there is no variability in their values. Hence their variances are always zero.

$\endgroup$
  • $\begingroup$ Yes, I get that part. So give me an example based on a dataset. And why is it that if the vectors are equal in length we can always get the cov regardless of whether we consider the vector random or not? $\endgroup$ – Antoni Parellada Oct 30 '15 at 4:12
  • 1
    $\begingroup$ A good way to see why this is the case is exactly as you have mentioned in your post. $E(X)=X, so X-E(X)=0\implies$ a zero covariance. You are able to calculate the covariance in R because it is $assuming$ you have passed it random vectors. Did you notice you didn't have to tell R whether your data was random or not? $\endgroup$ – StatsStudent Oct 30 '15 at 4:20
  • $\begingroup$ So it is not a numeric result... given two equal vectors we'll always be able to calculate the covariance; however, the result only makes sense if the vector is "random"? $\endgroup$ – Antoni Parellada Oct 30 '15 at 4:25
  • $\begingroup$ You can calculate the covariance of the returns on GE stock and Siemens over the course of one year. Yet, each one of these two vectors is a random variable, and not a random vector - a random vector composed of a list of random variables... $\endgroup$ – Antoni Parellada Oct 30 '15 at 4:29
  • $\begingroup$ Right, so all of this really rests on how you decide to treat the the vector. Depending on how you are using the vector, one might choose to treat the vector as random or constant. If you are treating the vector as a constant, the covariance is zero. Otherwise, you can calculate the covariance. $\endgroup$ – StatsStudent Oct 30 '15 at 4:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.