6
$\begingroup$

Questions:

  1. Can we talk about:
    variance of a deterministic variable?;
    covariance between a deterministic variable and a stochastic variable?;
    covariance between two deterministic variables?
  2. Are these concepts well defined in sample?; in population?

Motivation
Take a simple regression

$$y = \beta_0 + \beta_1 x + \varepsilon.$$

Suppose the regressor $x$ is stochastic. The OLS estimate of $\beta_1$ will be

$$\hat{\beta}_1=\frac{\widehat{\text{Cov}}(x,y)}{\widehat{Var}(x)}$$

where hats denote sample counterparts of the population concepts. No problem here.

Now suppose $x$ is deterministic. I am not sure if I can use terms like variance and covariance in this context. Should I exchange $\hat{\beta}_1=\frac{\widehat{\text{Cov}}(x,y)}{\widehat{Var}(x)}$ for something like

$$\hat{\beta}_1=\frac{\frac{1}{n-1}\sum(x_i-\bar{x})(y_i-\bar{y})}{\frac{1}{n-1}\sum(x_i-\bar{x})^2}$$

to be correct? But then again, how meaningful is $\bar{x}$ when $x$ is deterministic? So should I go all the way to

$$\hat{\beta}_1=\frac{\frac{1}{n-1}\sum_{i=1}^n(x_i-\frac{1}{n}\sum_{j=1}^n x_j)(y_i-\frac{1}{n}\sum_{j=1}^n y_j)}{\frac{1}{n-1}\sum(x_i-\frac{1}{n}\sum_{j=1}^n x_j)^2}?$$

I am picking on details here and this may not be too important; my main questions are listed at the top of the post.

$\endgroup$
6
$\begingroup$

All five questions have "yes" answers--but we have to be careful about what they mean.

  1. "Variance of a deterministic variable."

    Let's understand a "deterministic variable" to be a univariate dataset. It's just a bunch of values $X=x_1, x_2, \ldots, x_n$, with no probability model. By definition its variance is

    $$\text{Var}(X) = \frac{1}{n}\sum_{i=1}^n \left(x_i - \bar X\right)^2$$

    where $$\bar X = \frac{1}{n}\sum_{i=1}^n x_i$$ is its mean. There is no justification whatsoever to use $n-1$ instead of $n$ in any of these fractions--and this is never legitimately done--because no estimates are being made.

    We may always think of $X$ as defining a "population." This is the definition of a population variance.

  2. "Covariance between a deterministic variable and a stochastic variable."

    One way to understand this is to assume it refers to a sequence of the form $(x_1, Y_1), (x_2,Y_2), \ldots, (x_n,Y_n)$ where the $x_i$ are numbers and the $Y_i$ are random variables. Then we may define the random variable $$\bar Y = \frac{1}{n}\sum_{i=1}^n Y_i,$$ via which the covariance of $x$ and $Y$ is defined as

    $$\text{Cov}(x,Y) = \frac{1}{n}\sum_{i=1}^n (x_i - \bar x)(Y_i - \bar Y).$$

    It is a linear combination of the $Y_i$ and consequently is itself a random variable. This notation is frequently used as a shorthand in linear regression calculations.

  3. "Covariance between two deterministic variables."

    "Two deterministic variables" can be considered a dataset of ordered pairs $(x_1, y_1), (x_2,y_2), \ldots, (x_n,y_n)$. The covariance can be defined exactly as in (2) and interpreted similarly. In fact, this is a direct consequence of (1): after all, covariances are variances.

  4. "Are these concepts well defined in samples?"

    Because they are well-defined for any dataset, they are well-defined for a sample. Note that similar expressions with $n-1$ in the (outer) denominator are estimators: they are not the sample variance or sample covariance.

  5. "Are these concepts well defined in populations?"

    Because they are well-defined for any dataset, and a population can be considered a dataset (when fully enumerated), they are well-defined for a population.

$\endgroup$
1
$\begingroup$

The simple answer to your first three questions is no: it makes no sense in general to talk about variance or covariance involving a deterministic variable.

However, if we begin with a deterministic variable but then use some method of imposing a probability distribution on it, it then becomes a random variable, and the concept of variance then makes sense. For example, any deterministic variable can become a random variable simply by imposing a degenerate (i.e., constant) distribution on it; in this case, the variance (along with its covariance with any other random variable) becomes 0.

A more interesting way of imposing a distribution on deterministic random variables is to use the empirical distribution, based on an observed sample. That is, if you observe $x_1,\dots,x_n$ in a sample, then we can define a discrete probability distribution on $x$ by $$P(x=x_0)=\frac1n \cdot\text{the number of $i$ such that $x_i=x_0$}$$ for all $x_0$. For example, in the case where $x_1,\dots,x_n$ are all distinct, we get $P(x=x_i)=\frac1n$ for each $i=1,\dots,n$. If we use this probability distribution on $x$, then the mean of $x$ becomes simply the sample mean $\overline x=\frac1n\sum_{i=1}^n x_i$, and the variance of $x$ becomes $\hat\sigma^2=\frac1n\sum_{i=1}^n (x_i-\overline x)^2$.

This same idea can be applied in the situation where we have observed vectors $(x_1,y_1),\dots,(x_n,y_n)$. We can define a joint probability distribution on $(x,y)$ again by using the empirical distribution, and the covariance between $x$ and $y$ then becomes $$\text{Cov}(x,y)=\frac1n\sum_{i=1}^n (x_i-\overline x)(y_i-\overline y)$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.