# Concepts of Probability all messed up

I am having a really hard time taking concepts of probability space, experiment, random variables and stitching them together to make a robust understanding of the probability theory.

So every probability theory starts with an experiment. According to Wikipedia "In probability theory, an experiment or trial is any procedure that can be infinitely repeated and has a well-defined set of possible outcomes, known as the sample space".

Now, lets say, our experiment is to go to the library and see the first book that was issued that day. This can either be done just for a day (although of no significance from the point of statistical analysis) or few days. Suppose we go to the library for N days. These repetitions of the experiment are called trials, and these trials make up a composed experiment (or experiment). People sometimes use the term trials and experiment interchangeably or composed experiment and experiment interchangeably. But it's better to think of individuals trials within a single experiment (trials != experiment and composed experiment== experiment).

Now for the above experiment all the books in the library is our sample space. The random variables can be

1. No. of pages in the book (Let's say $X_{1}$)
2. Price of the book ($X_{2}$)
3. No. of authors ($X_{3}$)
4. No. of times the book has been issued before ($X_{4}$)

Now comes the concept of probability distribution of these random variables. Does these distributions and the entire Probability space always come from the experiment or even before we perform the experiment we have a theoretical of these variable ?

On our very first trial , let's say, we find that "The elements of Statistical Learning" is the book issued that day.

Let's assume

$$X_1 = \text{426 pages}$$ $$X_2 = \text{30 dollars}$$ $$X_3 = \text{3 authors}$$ $$X_4 = \text{Issued 50 times before}$$

With every other trial of our experiment we are changing out Probability Space (Sample Space, Set of Events, Probability measure)

So the previous definitions of Random Variables falls flat. Because now that we have a new sample space and Random Variables map outcomes in Sample Space to real numbers, we need new definitions of our random variables every time we perform one more trial. Or we could have definitions which are not influenced by these changes such as

1. No. of pages in the latest book ($Y_{1}$)
2. Price of the latest book ($Y_{2}$)
3. No. of authors of the latest book($Y_{3}$)
4. No. of times the latest book has been issued before ($Y_{4}$)

After performing $N$ trials we will end up with

$$Y_{1} = (y_{11}, y_{12}, y_{13}, \ldots , y_{1N})$$

$$Y_{2} = (y_{21}, y_{22}, y_{23}, \ldots , y_{2N})$$

$$Y_{3} = (y_{31}, y_{32}, y_{33}, \ldots , y_{3N})$$

$$Y_{4} = (y_{41}, y_{42}, y_{43}, \ldots , y_{4N})$$

The probability distributions of these random variables can now be used for statistical testing against true distributions. But again I am confused here, do these true distributions exist ? When one speaks of probability distributions of random variables is he talking about distributions calculate through $N$ trials or even before the experiment is done, that is, while defining probability space at the very beginning do we have a true theoretical probability distribution.

Now comes the question of Independent and Identically distributed random variables. The above random variables ($Y_{i}$) can be assumed as independent (debatable) but I cannot see how random variables here can be identically distributed. They dont even have the same range, Pages can take value from 0-3000 (let's say) , authors can never have such high numbers. So how can we ever assume that random variables are identically distributed ?

Speaking of Central Limit Theorem, it asks us to sum independent and identically distributed random variables but how can we add two different things, to me it seems like adding apples to oranges, No. of pages + Price of the book, doesn't make any sense ? Or are they asking us to add the elements of a single Random Variable($y_{i1}+y{i2}+y_{i3}+y_{i4}+\cdots+ y_{iN}$) but this still feels inconsistent to the definition of CLT which talks about sum of Independent and Identically Distributed Random variables.

One major reasons these doubts stem is because many books use some terms interchangeably like Random variables as a function from sample space to real space and Instances of Random Variables are both termed as Random variables, some never differentiate Probability Space from Real Space, events from random variables etc. Also the conventional dice throw and coin toss examples are way too simple and I feel do a pretty bad job of explaining these concepts and extrapolating them to real world examples. I would really appreciate if someone could make these concepts clearer to me using the example that I used above.

Using your example, you have four random variables $Y_1,Y_2,Y_3,Y_4$ (for number of pages, price, etc.). The variables are obviously not independent and not identically distributed, what you already noticed. Taking as example number of pages, this variable is defined in terms of probability distribution that describes probabilities of observing any possible number of pages in any possible book. The distribution is a function that maps number of pages to probability of observing book with such number of pages in your experiment. Drawing more samples form the distributions (more trials) does not change a thing about those random variables, you don't need "new definitions". In your example you observed $N$ draws from each of the four random variables. You ended up with observing $N\times 4$ random variables, e.g. $Y_{11}$ in your example is number of pages of the book observed on the first day etc. The $Y_{11}, Y_{21}, \dots, Y_{N1}$ random variables are independent and identically distributed. The $Y_{i1}$ is a random variable because you observed a random book on the $i$-th day, while the observed outcome $y_{i1}$ is not random, it is a known value. We use the observed outcomes to learn something about the unknown and unobservable random variables, this is where we use statistics. I can't see any difference between $Y_1$ and $X_1$ in your description, unless you use $X_1$ to denote any possible book, and $Y_1$ to denote some subset of $X_1$.
• Thanks for your reply. I guess one of the confusion I had was between Random Variables as a function that maps sample space to real numbers and Instances of random variables which are actually real numbers. Random variables is used interchangeably to mean either of the two. So when one says random variables are iid they aren't referring to two functions ($Y_{1}$ and $Y_{2}$ ) being iid but rather the instances of random variables ($y_{1i}$ ) being iid –  redenzione11 Oct 1 '17 at 20:37
• Could you please explain what $Y_{ij}$ (capital Y with two subscripts) is in your answer ? I never defined $Y_{ij}$ , just $Y_{i}$ . The one with two subscripts were small y $y_{ij}$ indicating an instance of random variable $Y_{i}$ on the jth day. –  redenzione11 Oct 1 '17 at 20:42
• @redenzione11 obviously two different functions are not identical! The two random variables $Y_{11}$ and $Y_{21}$ are i.i.d. since those are two random books, they share same distribution of the number of pages. I don't know what you mean by "instances" of random variables. If you plan to take a random book on $i$-th day and on $j$-th day, then there will be two random variables $Y_{i1}$ and $Y_{j1}$, while $y_{i1}, y_{j1}$ are the observed outcomes (this is how the notation us usually used). – Tim Oct 1 '17 at 20:45
• I am really sorry if I am using wrong notations or terms. By instances I mean observed values i.e realizations from a random variable. Okay so let me explain what I ve understood so far. First, Random Variables are functions that map Sample Space to real numbers, lets say $Y_{1}$ maps all books to their respective total number of pages. Now we can have many trials in an experiment and $Y_{1i}$ and $Y_{1j}$ are number of pages of the book on day $i$ and day $j$ , but since we haven't performed the trials yet we do not know their exact values hence random variables. –  redenzione11 Oct 1 '17 at 21:56
• And $y_{1i}$ and $y_{1j}$ are the observed values after we perform the trials. So whenever we hear IID random variables it's $Y_{1i}$ and $Y_{1j}$ that are independent and identically distributed (because we havent performed the experiment yet and know nothing of them expect that they are iid) and not $Y_{1}$ and $Y_{2}$ that are identically distributed. –  redenzione11 Oct 1 '17 at 22:00