7
$\begingroup$

Questions:

  1. Can we talk about:
    variance of a deterministic variable?;
    covariance between a deterministic variable and a stochastic variable?;
    covariance between two deterministic variables?
  2. Are these concepts well defined in sample?; in population?

Motivation
Take a simple regression

$$y = \beta_0 + \beta_1 x + \varepsilon.$$

Suppose the regressor $x$ is stochastic. The OLS estimate of $\beta_1$ will be

$$\hat{\beta}_1=\frac{\widehat{\text{Cov}}(x,y)}{\widehat{Var}(x)}$$

where hats denote sample counterparts of the population concepts. No problem here.

Now suppose $x$ is deterministic. I am not sure if I can use terms like variance and covariance in this context. Should I exchange $\hat{\beta}_1=\frac{\widehat{\text{Cov}}(x,y)}{\widehat{Var}(x)}$ for something like

$$\hat{\beta}_1=\frac{\frac{1}{n-1}\sum(x_i-\bar{x})(y_i-\bar{y})}{\frac{1}{n-1}\sum(x_i-\bar{x})^2}$$

to be correct? But then again, how meaningful is $\bar{x}$ when $x$ is deterministic? So should I go all the way to

$$\hat{\beta}_1=\frac{\frac{1}{n-1}\sum_{i=1}^n(x_i-\frac{1}{n}\sum_{j=1}^n x_j)(y_i-\frac{1}{n}\sum_{j=1}^n y_j)}{\frac{1}{n-1}\sum(x_i-\frac{1}{n}\sum_{j=1}^n x_j)^2}?$$

I am picking on details here and this may not be too important; my main questions are listed at the top of the post.

$\endgroup$

2 Answers 2

9
$\begingroup$

All five questions have "yes" answers--but we have to be careful about what they mean.

  1. "Variance of a deterministic variable."

    Let's understand a "deterministic variable" to be a univariate dataset. It's just a bunch of values $X=x_1, x_2, \ldots, x_n$, with no probability model. By definition its variance is

    $$\text{Var}(X) = \frac{1}{n}\sum_{i=1}^n \left(x_i - \bar X\right)^2$$

    where $$\bar X = \frac{1}{n}\sum_{i=1}^n x_i$$ is its mean. There is no justification whatsoever to use $n-1$ instead of $n$ in any of these fractions--and this is never legitimately done--because no estimates are being made.

    We may always think of $X$ as defining a "population." This is the definition of a population variance.

  2. "Covariance between a deterministic variable and a stochastic variable."

    One way to understand this is to assume it refers to a sequence of the form $(x_1, Y_1), (x_2,Y_2), \ldots, (x_n,Y_n)$ where the $x_i$ are numbers and the $Y_i$ are random variables. Then we may define the random variable $$\bar Y = \frac{1}{n}\sum_{i=1}^n Y_i,$$ via which the covariance of $x$ and $Y$ is defined as

    $$\text{Cov}(x,Y) = \frac{1}{n}\sum_{i=1}^n (x_i - \bar x)(Y_i - \bar Y).$$

    It is a linear combination of the $Y_i$ and consequently is itself a random variable. This notation is frequently used as a shorthand in linear regression calculations.

  3. "Covariance between two deterministic variables."

    "Two deterministic variables" can be considered a dataset of ordered pairs $(x_1, y_1), (x_2,y_2), \ldots, (x_n,y_n)$. The covariance can be defined exactly as in (2) and interpreted similarly. In fact, this is a direct consequence of (1): after all, covariances are variances.

  4. "Are these concepts well defined in samples?"

    Because they are well-defined for any dataset, they are well-defined for a sample. Note that similar expressions with $n-1$ in the (outer) denominator are estimators: they are not the sample variance or sample covariance.

  5. "Are these concepts well defined in populations?"

    Because they are well-defined for any dataset, and a population can be considered a dataset (when fully enumerated), they are well-defined for a population.

$\endgroup$
10
  • $\begingroup$ Can you please explain how we would use these definitions to calculate the correlation between $X$ and $Y$ when $(X_i, Y_i)=(Z_1+i, Z_2+i)$ with $Z_1, Z_2 \overset{\mathrm{i.i.d.}}{\sim}\mathcal{N}(0, 1)$ for deterministic $i=1,\ldots,n$? $\endgroup$
    – statmerkur
    Jan 20, 2022 at 0:07
  • $\begingroup$ @Statmerkur The bilinearity of covariance implies the addition of $i$ does not change the covariance, while the independence of the $Z_i$ implies that covariance must either be $0$ or undefined. Because the $Z_i$ have finite variances, the covariance will be $0.$ $\endgroup$
    – whuber
    Jan 20, 2022 at 4:46
  • $\begingroup$ Thank you. However, if I understand you correctly, the correlation estimate in this little R simulation n <- 100; cor(rnorm(n) + 1:n, rnorm(n) + 1:n) should be close to $0$ -- but it is actually close to $1$. The latter does make sense to me as the two sequences are identical up to a small noise component. Am I getting something wrong here? $\endgroup$
    – statmerkur
    Jan 20, 2022 at 15:09
  • $\begingroup$ @stat You don't understand correctly if you expect that correlation to be near $0,$ but I cannot guess what the nature of the misunderstanding might be. $\endgroup$
    – whuber
    Jan 21, 2022 at 14:44
  • $\begingroup$ I don't expect that correlation to be near $0$. Rather, I'm interested in how one would calculate the expected correlation in such a case. I was then confused about what you said above since, I thought, by the same reasoning the correlation between $X$ and $Y$ when $(X_i, Y_i)= (Z_{1,i}+i,Z_{2,i}+i)$ with $Z_{1,i}, Z_{2,i}\overset{\mathrm{i.i.d.}}{\sim}\mathcal{N}(0,1);i=1,\ldots,100$ would also have to be $0$. And I think the R code calculates the correlation in one particular realization of this sequence. $\endgroup$
    – statmerkur
    Jan 21, 2022 at 15:53
1
$\begingroup$

The simple answer to your first three questions is no: it makes no sense in general to talk about variance or covariance involving a deterministic variable.

However, if we begin with a deterministic variable but then use some method of imposing a probability distribution on it, it then becomes a random variable, and the concept of variance then makes sense. For example, any deterministic variable can become a random variable simply by imposing a degenerate (i.e., constant) distribution on it; in this case, the variance (along with its covariance with any other random variable) becomes 0.

A more interesting way of imposing a distribution on deterministic random variables is to use the empirical distribution, based on an observed sample. That is, if you observe $x_1,\dots,x_n$ in a sample, then we can define a discrete probability distribution on $x$ by $$P(x=x_0)=\frac1n \cdot\text{the number of $i$ such that $x_i=x_0$}$$ for all $x_0$. For example, in the case where $x_1,\dots,x_n$ are all distinct, we get $P(x=x_i)=\frac1n$ for each $i=1,\dots,n$. If we use this probability distribution on $x$, then the mean of $x$ becomes simply the sample mean $\overline x=\frac1n\sum_{i=1}^n x_i$, and the variance of $x$ becomes $\hat\sigma^2=\frac1n\sum_{i=1}^n (x_i-\overline x)^2$.

This same idea can be applied in the situation where we have observed vectors $(x_1,y_1),\dots,(x_n,y_n)$. We can define a joint probability distribution on $(x,y)$ again by using the empirical distribution, and the covariance between $x$ and $y$ then becomes $$\text{Cov}(x,y)=\frac1n\sum_{i=1}^n (x_i-\overline x)(y_i-\overline y)$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.