Chapter 5.5 (Page 130) of the Deep Learning Book gives formula (5.61)

enter image description here

for training machine learning models.

where $p_{model}(\mathbf x)$ is a parametric family of probability distributions over the same space indexed by $\mathbf{\theta}$

and $\hat p_{data}$ denotes the empirical distribution.

I'm conscious that the true probability $p_{data}(\mathbf x)$ is usually unknown in practice though, to make the question concrete and easy to understand, consider this true data-generating distribution

$x \sim \mathcal{N}_{true}(x; \mu, \sigma^2, \epsilon), \ where\ \mu=1, \sigma=0.5 \tag{1}$

and $\epsilon$ is drawn from another normal distribution with a mean of zero and standard deviation of 0.01.

let $ \mathbb D = \{x^{(1)}, \cdots, x^{(m)}\} $ denotes the set of m=30 training examples drawn independently from formula (1).

This Python code is to simulate the data generation

# m denotes the number of training examples
m_examples = 30
noises = np.random.normal(0, .01, m_examples)
D_dataset = np.random.normal(1, .5, m_examples) + noises

and, here are the training examples in $\mathbb D$

array([1.09511424, 1.19308283, 0.56589451, 0.0320107 , 0.84471951,
       1.06840171, 1.62464622, 1.59967635, 0.8053044 , 0.85295461,
       0.47716395, 0.30453377, 0.15447528, 1.97660445, 0.74961254,
       0.78429959, 0.38854311, 1.3866936 , 0.19618175, 0.8850889 ,
       0.52673682, 1.19998743, 0.75324179, 0.40226226, 1.00860643,
       1.19962228, 1.0337162 , 1.14936411, 0.69816675, 0.833323  ])

the arithmetic mean

>>> np.mean(D_dataset)

the standard deviation

>>> np.sqrt(np.var(D_dataset))

Given that, is the distribution

$x \sim \mathcal{N}_{true}(x; \mu, \sigma^2), \ where\ \mu=0.85966, \sigma=0.44726 \tag{2}$

the empirical distribution $\hat p_{data}$, the estimator p_{model} or something else?

The part of the book cited above considers $\hat p_{data}$ a function only of the data-generating process, what does that mean?


1 Answer 1


$\newcommand{\model}{p_{\text{model}}}\newcommand{\emp}{\hat p_\text{data}}$The empirical distribution is a discrete distribution that puts a mass of $1/n$ on each realized data point (if there are no ties; if $x_i$ appears $m$ times then its mass will be $m/n$). $\emp$ does not depend on the model at all. The data generating process completely determines what $\emp$ will be. For example, if we have $X_1, \dots, X_n \stackrel{\text{iid}}\sim \mathcal N(\mu_0, 1)$, then that already completely determines the behavior of $\emp$. It doesn't matter how we're modeling this or what our estimate of $\mu_0$ is.

It might help to write out $\emp$ more fully as $$ \emp(x) = \frac 1n \sum_{i=1}^n \mathbf 1_{X_i = x} $$ so we can see that $\emp$ is a random variable itself (technically it is a Radon-Nikodym derivative of a random measure) and its distribution is completely induced by the distribution of the $X_i$. $\emp$ does depend on $\mu_0$, the true parameter, since that's what specifies the particular data generating process of $\mathcal N(\mu_0, 1)$, but our model does not affect this and $\mu_0$ is also constant here.

And $\text E_{\emp}[\log \emp(x)]$ is even easier: this is the negative entropy of a discrete uniform distribution over $\{x_1, \dots, x_n\}$ so we have $$ \text E_{\emp}[\log \emp(x)] = \frac 1n \sum_{i=1}^n \log \emp(x_i) = \frac 1n \sum_i \log\left(\frac 1n\right) = -\log n. $$

That's why, from the perspective of fitting $\model$, $\text E_{\emp}[\log \emp(x)]$ is constant since it does not vary with the model parameters $\theta$.

  • $\begingroup$ Your answer is quit understandable. Thank you so much! Does $\mathcal N(\mu_0, 1)$ means the standard deviation of the normal distribution $\sigma=1$, is a known quantity in this example? Is my understanding correct? $\endgroup$
    – soplus2018
    Sep 1, 2021 at 0:02
  • $\begingroup$ @soplus2018 hey, yeah, that's correct. The variance parameter wasn't important for my example so I just fixed it at a known value of $1$. $\endgroup$
    – jld
    Sep 10, 2021 at 18:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.