3
$\begingroup$

I'am currently following a basic statistics course and a machine learning course. I try to understand what is covariance. In general, books define covariance as follows: Covariance is a measure of how much two random variables vary together. Okay, I got the definition, I understood the formula of covariance etc. However, when I search on the web to find an example where covariance calculated, I found a video on youtube. Starting from 43 seconds in that video, she tries to find the covariance of set of 4 data points. This is the part I confused. How the random variables are related with data points ? Is it because these data points are sampled from a distribution we don't know and if we talk about a distribution there have to be a random variable ? If this is the case, isn't it a continuous random variable ? Is it possible to calculate the covariance of 2 continuous random variables in the same way as it is done for discrete random variables ?

$\endgroup$
  • 2
    $\begingroup$ Nothing in the universe is inherently a random variable: calling a quantity a random variable is a way of announcing a way of modeling the world. I posted a non-technical explanation of this at stats.stackexchange.com/a/54894/919. $\endgroup$ – whuber Dec 7 '18 at 13:32
3
$\begingroup$

Let $X$ and $Y$ be two real random variables defined on a probability space $(\Omega,\mathcal{F},\mathbb{P})$, where $\Omega$ is a set of outcomes $\omega$. They then also can be seen as functions $X\colon \Omega \to \mathbb{R}$ and $Y\colon \Omega \to \mathbb{R}$. That is, e.g., $X(\omega)\in \mathbb{R}$ for some $\omega\in\Omega$ is a number, called a realization of $X$, and that is precisely what we observe in practice. (That is, data points are realizations of random variables, contrary to your question title.)

As an example, let $X$ and $Y$ be a person's height and weight, and let $X_1, Y_1, \dots, X_n, Y_n$ be independent and identically distributed copies of $X$ and $Y$, so that we are dealing with $n$ people. We may be interested in knowing $$ \operatorname{\mathbb{C}ov}[X,Y]=\mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y]. $$ While $X$ and $Y$ indeed are random variables (or functions) that we do not expect to fully ever now, in practice we may observe realizations $X_1(\omega), Y_1(\omega), \dots, X_n(\omega), Y_n(\omega)$. Now it is natural to expect that, if $n$ is large enough, we may get a pretty good idea about $X$ and $Y$.

For instance, if you knew that those $n$ people in theory are equally likely to be of any height $h$ (i.e., they heights are identically distributed), then after knowing $X_1(\omega), \dots, X_n(\omega)$ you probably would be quite confident that $\mathbb{E}[X]$ and $n^{-1}\sum_{i=1}^n X_i(\omega)$ are really close.

The same happens with covariance. When $n$ is large enough and certain regularity conditions hold, using various law of large numbers we get that those empirical estimates $\widehat{\operatorname{\mathbb{C}ov}}_n[X,Y]$ that we compute in practice do indeed come close to the true $\operatorname{\mathbb{C}ov}[X,Y]$.

$\endgroup$
  • $\begingroup$ thank you for your response. The thing I am not able to understand is how a person's weight/height can be a random variable ? I guess my problem is there. I mean, I can understand rolling a dice can be a random variable but I don't understand this height/weight case. Is it possible to explain that part with simpler terms ? $\endgroup$ – zwlayer Dec 7 '18 at 13:21
  • $\begingroup$ @zwlayer, reading wikipedia articles on probability spaces and random variables might help. Imagine that $\Omega$ is a set of universes and $\omega\in\Omega$ is a universe. Then everything in our lives is a result/realization of a single "life experiment" where $\omega\in\Omega$ was chosen long time ago. When rolling a dice, you may think that $\omega\in\Omega$ is chosen at that moment. $\endgroup$ – Julius Vainora Dec 7 '18 at 13:28
  • $\begingroup$ @zwlayer, see also stats.stackexchange.com/questions/50/… and math.stackexchange.com/questions/240673/… $\endgroup$ – Julius Vainora Dec 7 '18 at 13:34
  • $\begingroup$ thank you for the resources. One more question: I read the sample space wikipedia article you shared and based on the definitions on that page, can we say that the experiment in your example (Random variable X) getting a height of a person ? In that case sample space is the all positive real numbers right ? And what is the ω in that example ? Is it some integer defining hight of a person ? Is it some kind of ID assigned to each person ? $\endgroup$ – zwlayer Dec 7 '18 at 13:52
  • $\begingroup$ @zwlayer, there is no single way to define such things; as I said, we can imagine that everything we observe (including a dice roll) is one big result of a single large experiment about our whole universe. However, the sample space that you suggest is indeed sensible, the meanings of $\omega$ as well. As for the experiment part (when it happens, what it is), I guess it's somewhat philosophical and depends on what you mean by "getting". Coin toss or dice rolls are much better for such examples since their realization "appears" only in front of us, rather than $n$ years ago... $\endgroup$ – Julius Vainora Dec 7 '18 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.