# The Deep Learning Book considers the empirical distribution $\hat p_{data}$ a function only of the data-generating process, what does that mean?

Chapter 5.5 (Page 130) of the Deep Learning Book gives formula (5.61)

for training machine learning models.

where $$p_{model}(\mathbf x)$$ is a parametric family of probability distributions over the same space indexed by $$\mathbf{\theta}$$

and $$\hat p_{data}$$ denotes the empirical distribution.

I'm conscious that the true probability $$p_{data}(\mathbf x)$$ is usually unknown in practice though, to make the question concrete and easy to understand, consider this true data-generating distribution

$$x \sim \mathcal{N}_{true}(x; \mu, \sigma^2, \epsilon), \ where\ \mu=1, \sigma=0.5 \tag{1}$$

and $$\epsilon$$ is drawn from another normal distribution with a mean of zero and standard deviation of 0.01.

let $$\mathbb D = \{x^{(1)}, \cdots, x^{(m)}\}$$ denotes the set of m=30 training examples drawn independently from formula (1).

This Python code is to simulate the data generation

# m denotes the number of training examples
m_examples = 30
np.random.seed(0)
noises = np.random.normal(0, .01, m_examples)
D_dataset = np.random.normal(1, .5, m_examples) + noises


and, here are the training examples in $$\mathbb D$$

array([1.09511424, 1.19308283, 0.56589451, 0.0320107 , 0.84471951,
1.06840171, 1.62464622, 1.59967635, 0.8053044 , 0.85295461,
0.47716395, 0.30453377, 0.15447528, 1.97660445, 0.74961254,
0.78429959, 0.38854311, 1.3866936 , 0.19618175, 0.8850889 ,
0.52673682, 1.19998743, 0.75324179, 0.40226226, 1.00860643,
1.19962228, 1.0337162 , 1.14936411, 0.69816675, 0.833323  ])


the arithmetic mean

>>> np.mean(D_dataset)
0.8596676363956413


the standard deviation

>>> np.sqrt(np.var(D_dataset))
0.44725697845961726


Given that, is the distribution

$$x \sim \mathcal{N}_{true}(x; \mu, \sigma^2), \ where\ \mu=0.85966, \sigma=0.44726 \tag{2}$$

the empirical distribution $$\hat p_{data}$$, the estimator p_{model} or something else?

The part of the book cited above considers $$\hat p_{data}$$ a function only of the data-generating process, what does that mean?

$$\newcommand{\model}{p_{\text{model}}}\newcommand{\emp}{\hat p_\text{data}}$$The empirical distribution is a discrete distribution that puts a mass of $$1/n$$ on each realized data point (if there are no ties; if $$x_i$$ appears $$m$$ times then its mass will be $$m/n$$). $$\emp$$ does not depend on the model at all. The data generating process completely determines what $$\emp$$ will be. For example, if we have $$X_1, \dots, X_n \stackrel{\text{iid}}\sim \mathcal N(\mu_0, 1)$$, then that already completely determines the behavior of $$\emp$$. It doesn't matter how we're modeling this or what our estimate of $$\mu_0$$ is.
It might help to write out $$\emp$$ more fully as $$\emp(x) = \frac 1n \sum_{i=1}^n \mathbf 1_{X_i = x}$$ so we can see that $$\emp$$ is a random variable itself (technically it is a Radon-Nikodym derivative of a random measure) and its distribution is completely induced by the distribution of the $$X_i$$. $$\emp$$ does depend on $$\mu_0$$, the true parameter, since that's what specifies the particular data generating process of $$\mathcal N(\mu_0, 1)$$, but our model does not affect this and $$\mu_0$$ is also constant here.
And $$\text E_{\emp}[\log \emp(x)]$$ is even easier: this is the negative entropy of a discrete uniform distribution over $$\{x_1, \dots, x_n\}$$ so we have $$\text E_{\emp}[\log \emp(x)] = \frac 1n \sum_{i=1}^n \log \emp(x_i) = \frac 1n \sum_i \log\left(\frac 1n\right) = -\log n.$$
That's why, from the perspective of fitting $$\model$$, $$\text E_{\emp}[\log \emp(x)]$$ is constant since it does not vary with the model parameters $$\theta$$.
• Your answer is quit understandable. Thank you so much! Does $\mathcal N(\mu_0, 1)$ means the standard deviation of the normal distribution $\sigma=1$, is a known quantity in this example? Is my understanding correct? Sep 1, 2021 at 0:02
• @soplus2018 hey, yeah, that's correct. The variance parameter wasn't important for my example so I just fixed it at a known value of $1$.