7
$\begingroup$

I've been pondering the differences in notation / language used in some of the resources I've read for statistics / machine learning.

Warning: this might be embarrassingly obvious to any decent statistician, so apologies in advance.

1 iid data (Bayesian)

I am a big fan of Pattern Recognition for Machine Learning (Bishop) so much of my foundation in basic statistical machine learning comes from there. In the first chapter (section 1.2.4, page 26), he takes the approach of defining a scenario where we have a random variable $x$ and the data $\pmb{\mathsf{x}} = \{x_1, ..., x_n\}$ are $N$ independently drawn observations of the random variable $x$.

To emphasize...a set of data is seen as a number of observations of the same random variable. My claim is that this approach to setting up the data / statistical problem is kind of Bayesian and this is corroborated (somewhat) by the fact that Bishop seems to be a Bayesian.

This also seems Bayesian to me because we're setting the data in stone. That is, our data is not a set of random variables but a set of realizations of a single random variable which falls in line with Bayesian philosophy.

2 iid random variables (Frequentist)

Contrast this with a more frequentist resource such as Larry Wasserman's intermediate statistics notes. Throughout the lectures, we consider instead a set of random variables $X_1, ..., X_N$ that are i.i.d in that they are independent and follow the same exact distribution.

Data is seen as a set of random variables, not a set of realizations of a random variable. This lends itself to constructing "estimators" and asymptotics because we view the dataset as random. It can be constructed many times by realizing the set of random variables $X_1, ..., X_N$ simultaneously many times. Then you can do analysis of the long-run behaviour of doing an experiment many times which I believe is quite Frequentist.

I suppose my question is: what are the relative merits of using either approach to denoting the data? And does this directly coincide with the merits of Bayesianism vs Frequentism?? I am biased and prefer the Bayesian setup because (i) I'm used to it, and (ii) it allows (in my opinion) for much less "terminological" overhead that often creates abusive notation and then confusion. However, it does seem like there is a much easier segway into things like asymptotics (maybe other stuff I'm not mentioning) if we view the dataset as random.

Of course, please point out any errors! Thanks.

Edit: I defer to this explanation from Christian Hennig.

$\endgroup$
13
  • 1
    $\begingroup$ "a number of observations of the same random variable" this is not very clear and it sounds more like a play with words and definitions 'when do we call something a variable or not?' leading to semantic discussions where the underlying principles and concepts are not different but the one calls it a 'realization' and the other a 'variable'. I doubt whether you are describing an actual difference and some practical example to make it more clear might be useful. $\endgroup$ Jul 8, 2023 at 22:07
  • $\begingroup$ In my mind this means -- suppose you have a random variable $x$ that follows a univariate Gaussian distribution. You sample from this distribution a number $N$ times to yield $N$ observations of that random variable. An example could be as simple as the random variable representing height of humans on earth. Going out and recording $N$ measurements would yield $N$ observations of that random variable. $\endgroup$
    – paul
    Jul 9, 2023 at 3:29
  • 1
    $\begingroup$ Paul, in a frequentist approach, one can just as well think about data as realizations of some random process/variable. In your example, whether you call it realizations or random variables, the concepts are the same with only different labels/names. You have to be very precise here and describe what you exactly mean, in order to prevent to be describing some difference that doesn't actually exist and is just a use of different words. $\endgroup$ Jul 9, 2023 at 15:55
  • 1
    $\begingroup$ In a Bayesian setting, elements of a sample are conditionally independent and unconditionally exchangeable. The keyword exchangeability may be helpful when seeking the distinction between Bayesian and frequentist paradigms. $\endgroup$
    – Xi'an
    Jul 10, 2023 at 7:59
  • 1
    $\begingroup$ @paul: We cannot sample from a random variable $x$ $n$ times, I think that is where you go wrong. If we have $n$ observation, that is modelled as $n$ random variables, maybe iid $\endgroup$ Sep 4, 2023 at 18:50

4 Answers 4

4
$\begingroup$

There are no two definitions or usage of the term. The only valid way of understanding "i.i.d. data" is the second "frequentist", as you call it, way. The only thing that could have distributions that could be independent are random variables. The only things that could have conditional distribution, or joint distribution, that makes a likelihood function in a Bayesian setting, are random variables. It is not data that is independent and identically distributed, but the random variables that we consider the data to be the realizations of which.

On the referred page 26, the the author uses the following example to illustrate it:

Because our data set $\mathbf{x}$ is i.i.d., we can therefore write the probability of the data set, given $\mu$ and $\sigma^2$ in the form $$ p(\mathbf{x}|\mu,\sigma^2)=\prod_{i=1}^n \mathcal{N}(x_i|\mu,\sigma^2) $$

So each $x_i$ follows a normal distribution and their joint distribution is a product because $x_i$s are independent. They need to be random variables for that.

Also, notice that the Bayes theorem tells us that we can obtain

$$ p(\mu,\sigma^2|\mathbf{x}) \propto p(\mathbf{x}|\mu,\sigma^2)\, p(\mu)\, p(\sigma^2) $$

You can condition only on random variables, so if $\mathbf{x}$ would not be a random variable, the conditional probability would not be possible.

$\endgroup$
16
  • 2
    $\begingroup$ It could be added that in the frequentist as well as the Bayesian setup, once the data are observed, the values taken by them or realisations are no longer random variables but fixed, normally distinguished by lower case letters $x_1,\ldots,x_n$ from the random variables $X_1,\ldots,X_n$. $\endgroup$ Jul 8, 2023 at 22:21
  • $\begingroup$ Thank you for the response! Could you add what you think Bishop then means when he sets up his problems? He usually describes data as being $N$ observations of a random variable drawn independently from the distribution that random variable follows. He then calls the dataset "i.i.d" (See first paragraph of page 26 here). This seems in direct contrast to when you say "the only valid way of understanding "i.i.d" data is the second frequentist". Thank you! $\endgroup$
    – paul
    Jul 9, 2023 at 3:26
  • $\begingroup$ @paul I cannot talk for Bishop, but as I said, if you are not considering your data as random variables, you cannot have probabilistic models for them. Nonetheless, see the edit. $\endgroup$
    – Tim
    Jul 9, 2023 at 5:07
  • $\begingroup$ Thank you @Tim! I wanted to point our that your example does not faithfully represent what Bishop says, and this indeed lies at the heart of my question: Bishop uses a different boldface $\pmb{\mathsf{x}}$ in the likelihood $p(\pmb{\mathsf{x}}|\mu, \sigma^2)$. This $\pmb{\mathsf{x}}$, as I mentioned in the last comment, denotes (what Bishop says) is $N$ observations of a single random variable $x$. To me this makes sense because $\pmb{\mathsf{x}}$ is our dataset and is fixed which is true in practice. $\endgroup$
    – paul
    Jul 9, 2023 at 15:42
  • 1
    $\begingroup$ @Tim thank you for this! I agree with this statement. I was just confused because of the language Bishop uses. He seems to think that each toss is an observation of a single random variable. It would appear you simply disagree with him, and perhaps you're right! $\endgroup$
    – paul
    Jul 9, 2023 at 16:37
3
$\begingroup$

he takes the approach of defining a scenario where we have a random variable $x$ and the data $\pmb{\mathsf{x}} = \{x_1, ..., x_n\}$ are $N$ independently drawn observations of the random variable $x$.

What's in a name? That which we call $X_i \underset{iid }{\sim} N(0,1)$ by any other name would just be as random.

Call it 'multiple realizations of a variable' or 'multiple variables drawn from a distribution/population', it doesn't matter. Bishop doesn't define the problem/situation by calling it 'observations of a single variable' it is the name that he calls it that becomes defined as the situation that he describes.

$\endgroup$
1
$\begingroup$

After some back-and-forth with @Tim and @Sextus Empiricus (thank you both!) I think I have a clearer answer to my question:

First let's get notation down:

  • $X_i$ is a random variable.
  • $x_i$ is the realized value of observing/measuring $X_i$. It is a fixed number.
  • $\mathbf{X} = \{X_1, ..., X_N\}$ is a set of i.i.d random variables. This is a set of random variables.
  • $\pmb{\mathsf{x}} = \{x_1, ..., x_N\}$ is a set or realizations of the random variables $\{X_1, ..., X_n\}$. This is a set of fixed numbers.

It would seem that the following is technically incorrect (as Bishop does): "we have a random variable $X$ and the data $\pmb{\mathsf{x}} = \{x_1, ...,x_n\}$ are $N$ independently drawn observations of the random variable $X$". Once a random variable is observed, you cannot observe it again. So, technically speaking, this is does not make true mathematical sense.

The technically correct way to denote a fixed dataset (as Bayesians do) is to say: we have a set of realizations $\pmb{\mathsf{x}} = \{x_1, ..., x_N\}$ corresponding to observing the set of i.i.d random variables $\{X_1, ..., X_N\}$. A potential alternative to denote a fixed dataset (as Bayesian do) is to say: we have a set of values $\{x_1, ..., x_N\}$ that were all independently sampled from the same distribution $p(x)$.

What you'll often see in Frequentist texts is a dataset that is not yet realized (because they treat the dataset as random). Something like: we have a set of data comprised of i.i.d random variables $\{X_1, ..., X_N\}$. The benefit of this is that we can remark on properties of these data as they were random variables. For example, we can explore the properties of "estimators" which are functions of the random variables.

$\endgroup$
1
  • $\begingroup$ Exactly, +1! But bayesians also treat the sample as random, how else could they get a likelihood function? $\endgroup$ Sep 4, 2023 at 18:55
-4
$\begingroup$

Data are never "drawn from a distribution." Real data are outcomes from measurements, and hence ontological.

Probability distributions come from models; they exist only in our heads, hence epistemological. We can adopt one as a working hypothesis, but we are never certain whether they are true.

There are no random variables in Bayesian theory, only probability distributions.

However, when you adopt a probability distribution, of course, you can draw random samples from it, and analyze them as if they were data. But beware that these are not real data. Real data are different; they hide unexpected quirks.

Bishop's PRML is an excellent book for learning a lot of Bayesian methods. However, in Chapter 1, Bishop adopts a frequentist approach, which does a disservice to the remainder of his book.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.