Section 5.2 of the deep learning book says
How can we affect performance on the test set when we get to observe only the training set? The field of statistical learning theory provides some answers. If the training and the test set are collected arbitrarily, there is indeed little we can do. If we are allowed to make some assumptions about how the training and test set are collected, then we can make some progress.
The train and test data are generated by a probability distribution over datasets called the data generating process. We typically make a set of assumptions known collectively as the i.i.d. assumptions These assumptions are that the examples in each dataset are independent from each other, and that the train set and test set are identically distributed, drawn from the same probability distribution as each other. This assumption allows us to describe the data generating process with a probability distribution over a single example. The same distribution is then used to generate every train example and every test example. We call that shared underlying distribution the data generating distribution, denoted pdata. This probabilistic framework and the i.i.d. assumptions allow us to mathematically study the relationship between training error and test error.
can someone give some concrete examples to explain "a probability distribution over a single example" in "This assumption allows us to describe the data generating process with a probability distribution over a single example."
taking the example of the Bernoulli distribution, an output of head [1]
could be a single example?