3
$\begingroup$

I am taking an introductory course on statistics/probability and there's a concept that I am confused with.

That is the difference between empirical histogram vs PMF.

First off, let's use the example of rolling a fair dice. Then we do "secretly know" the underlying PMF is given by

$$ \mathbb{P}[X = i] = \frac{1}{6} \forall i \in \{1, 2, \ldots, 6\} $$

More concretely, the distribution above consists of the theoretical probability of each face. It is called a probability distribution and is not based on observed data. It can be studied and understood without any dice being rolled.

Empirical distributions, on the other hand, are distributions of observed data. They can be visualized by empirical histograms. https://inferentialthinking.com/chapters/10/1/Empirical_Distributions.html

But most books/courses will go on to use a simulation (either R or Python) to illustrate, for example, repeated sampling with replacement from 1-6 for $N$ number of times and show that as $N \to \infty$, the histogram generated by the sampling will converge to its true PMF (i.e. all $\frac{1}{6}$). I have no issue with this, but the underlying sampling with replacement uses a uniform distribution, and that means we are assuming that the underlying distribution is 1 out of 6 times, so it puzzles me on why the empirical histogram is not actually just generated from the PMF? What I envisioned is that I am given a dataset of say $N = 1000$ throws by people, and then perform statisical analysis to deduce that indeed the true PMF of a dice roll is $\dfrac{1}{6}$. I think I am missing a key point here.

Attached below is an image I found useful but unable to reconcile immediately on why using random sampling to generate observations does not implicitly assume the underlying probability distribution. enter image description here

EDIT: I think I understood the idea here, the simulating process is a didactical way of showing the convergence property. In more detail, let $X$ be a random variable for the relative frequency of landing on $6$, and let us be given a dataset full of dice rolls (1-6) of size $N = 10$, then say the dataset consists of only one $6$ out of the 10 data points, our relative frequency is $\frac{1}{10}$, this dataset is our empirical distribution (histogram), if we truly believe the theorem that the empirical distribution will converge to its true distribution, then repeating it enough times $N \to \infty$ will make our relative frequency of the dice face 6 to be $\frac{1}{6}$. To prove this point, the authors will have to show us first the true distribution and then simulate to show us it is indeed converging to the true distribution.

$\endgroup$
3
  • $\begingroup$ It's not clear what your question is. If someone uses simulation to show that it will lead to some distribution, then not using simulation will not show that. $\endgroup$
    – Tim
    Oct 2, 2022 at 5:08
  • $\begingroup$ I’m confused if using the simulation is the process of generating data or estimating model, as shown in the image above. To me it seems like generating data, but got confused because given enough trials, it can converge to the true distribution, which sounds like estimating the “model” arrow. $\endgroup$
    – nan
    Oct 2, 2022 at 5:32
  • $\begingroup$ It's an example to prove that the procedure works and it has nothing to do with estimating anything in real life where you don't know the true distribution. $\endgroup$
    – Tim
    Oct 2, 2022 at 5:54

1 Answer 1

7
$\begingroup$

What they want to demonstrate by using the simulations, is only that relative frequencies "converge" to the probabilities for $n\to\infty$. And for that, they do have to presume the probabilities, otherwise, they would not be able to check this convergence.

Once you have convinced yourself that lots of empirical trials do make the frequencies approach the probabilities, you can start doing experiments like the one that you have suggested with the 1000 actual throws. But they would be pointless if you wouldn't believe in this convergence. And, of course, you could (maybe with a little more than 1000 throws) discover that the die is not perfect and the probabilities are likely not to be all $\frac{1}{6}$.

The computer simulations could also be interpreted as testing the pseudo-random number generator for its quality, but again, only once convergence is accepted.

$\endgroup$
3
  • $\begingroup$ To clarify, is simulating the trials with the assumption of probability distribution a way of generating data or estimating / inferencing the true distribution, as shown in the diagram above. $\endgroup$
    – nan
    Oct 2, 2022 at 5:35
  • 2
    $\begingroup$ You usually do simulations to generate data that you then know the underlying probabilities of. Using this data next to in turn infer the original probabilities is only useful either for didactical reasons (as in the books you mentioned), to demonstrate the convergence of relative frequencies, or to test your simulation or your inference method. $\endgroup$
    – frank
    Oct 2, 2022 at 5:54
  • 1
    $\begingroup$ Thanks, I have made an edit to my question to replicate what you meant, do let me know if my idea is off... $\endgroup$
    – nan
    Oct 3, 2022 at 2:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.