# What does it mean for the training data to be generated by a probability distribution over datasets

I was reading the Deep Learning book and came across the following para (page 109, second para):

The training and test data are generated by a probability distribution over datasets called the data-generating process. We typically make a set of assumptions known collectively as the i.i.d. assumptions. These assumptions are that the examples in each dataset are independent from each other and that the training set and test set are identically distributed, drawn from the same probability distribution as each other. This assumption enables us to describe the data-generating process with a probability distribution over a single example.The same distribution is then used to generate every train example and every test example. We call that shared underlying distribution the data-generating distribution, denoted pdata. This probabilistic framework and the i.i.d. assumptions enables us to mathematically study the relationship between training error and test error.

Can somebody please explain me the meaning of this paragraph.

On page 122 the last para, it also gives an example of

a set of samples{x(1), . . . , x(m)} that are independently and identically distributed according to a Bernoulli distribution with mean θ.

What does this mean?

EDIT: To provide more details;

1) Probability distrubution over datasets: What are the datasets? How is the probability distribution generated?

2) The examples are independent of each other. Can you give me an example where the examples are dependent?

3) ..drawn from the same probability distribution as each other. Suppose the probability distribution is Gaussian. Does the term "Same probability distribution" mean that all the examples are drawn from a Gaussian distribution with same mean and variance?

4) ..This assumpition enables us... What does this mean?

5) Finally for the last para of page 122; it is given that the samples follow Bernoulli distribution. What does this mean intuitively?

• The third sentence in the paragraph you quote is the key one. Can you tell us more specifically what is unclear about it? – Stephan Kolassa Dec 26 '17 at 8:37
• @StephanKolassa Added details to the question. – ragvri Dec 26 '17 at 9:03
• Thank you. Your edits do clarify matters. It's still a rather broad question, but @sww's answer is already quite to the point. – Stephan Kolassa Dec 26 '17 at 9:37

1) Probability distrubution over datasets: What are the datasets? How is the probability distribution generated?

Once we can estimate the underlying distributions of the input data, we essentially know how they are picked and can do good predictions. [Generative]. Normally, we can assume an underlying distribution according to what we believe [inductive bias] For example, if we believe that there is a high probability that values are close to zero, we can take a Gaussian distribution with mean 0 and tune the parameters like variance when we train. Datasets are for example, set of all coin tosses and the distribution assumed will be binomial. When we do say maximizing log likelihood for the actual data points, we will get those parameters which make the dataset fit into the distribution assumed

2) The examples are independent of each other. Can you give me an example where the examples are dependent?

For example, we toss a coin and if we have a head we toss another otherwise we do not. Here there is a dependence between subsequent tosses

3) ..drawn from the same probability distribution as each other. Suppose the probability distribution is Gaussian. Does the term "Same probability distribution" mean that all the examples are drawn from a Gaussian distribution with same mean and variance?

and

4) ..This assumpition enables us... What does this mean?

Yes. That is why (4) is said. Once you have a probability distribution from one example, you do not need other examples to describe the data generating process.

5) It means that each example can be thought of as a coin toss . If the experiment was multiple coin tosses you would have each coin toss independent with probability of head to be .5 . Similarly if you choose any other experiment the result of each example can be thought of as a coin toss or a n dimensional dice if you will with each face corresponding to a result

Generating examples mean getting a distribution closest to what we see in the dataset for training. That is got by assuming a distribution and maximising likelihood of the given dataset and outputting the optimum parameters

• Thanks a lot. Can you please explain how is the training and test data generated from the data generated process? Once we have the probability distribution from one example, we know the probability distribution of other examples. But how are the examples actually generated? – ragvri Dec 26 '17 at 9:53
• @rjmessibarca It doesn't matter who or what generated the examples but it is the examples we are given or can get which are generated following some distribution known to us(optimization scenario) or unkown to us(machine learning issue). – Lerner Zhang Dec 1 '18 at 7:37