Can someone give some concrete examples to explain “a probability distribution over a single example” mean?

Question

Section 5.2 of the deep learning book says

How can we affect performance on the test set when we get to observe only the training set? The field of statistical learning theory provides some answers. If the training and the test set are collected arbitrarily, there is indeed little we can do. If we are allowed to make some assumptions about how the training and test set are collected, then we can make some progress.

The train and test data are generated by a probability distribution over datasets called the data generating process. We typically make a set of assumptions known collectively as the i.i.d. assumptions These assumptions are that the examples in each dataset are independent from each other, and that the train set and test set are identically distributed, drawn from the same probability distribution as each other. This assumption allows us to describe the data generating process with a probability distribution over a single example. The same distribution is then used to generate every train example and every test example. We call that shared underlying distribution the data generating distribution, denoted pdata. This probabilistic framework and the i.i.d. assumptions allow us to mathematically study the relationship between training error and test error.

can someone give some concrete examples to explain "a probability distribution over a single example" in "This assumption allows us to describe the data generating process with a probability distribution over a single example."

taking the example of the Bernoulli distribution, an output of head [1] could be a single example?

Answer 1

Here they want to refer to the fact that, if we assume that the sample observations are realizations of a iid random variable with a certain true probability distribution (described by its moments: mean, variance, and other moments), then we can attribute a probability to each observation in the sample (and to the sample as a whole aggregating the above mentioned probability for each observation). So if you assume that each observation has a certain distribution (like Bernoulli as you said), and observations are independent and identically distributed, then you know the probability for each observation in the sample (example), and you can use that distribution to calculate:

• the probability of having any given observation in your sample (since the obs are identically distributed)

• the probability of observing a certain sample by aggregating those probabilities for all the observations (since the observations are independent, you can take the product of the probability of each observation to get to the combined probability of having the set of observations in the sample).

In this sense a single example/observation with an assumed probability distribution allows you to model the probability for all the observation in the samples (as observations are assumed to be Independent and Identically distributed).

See the answers to this yesterday question for more details and a broader explanation.