Mathematical explanations of cross-entropy function I'm new to machine learning and know probability theory but most explanations of machine learning I found are very technically imprecise, and I'm having trouble understanding and describing machine learning concepts in a fully technically correct way that would satisfy a mathematician.
For example, I understand what the cross-entropy function does, but could someone explain and rewrite it using probability theory and information theory?
$$J = -\frac{1}{N} \sum_{n=0}^{N-1}\sum_{c=0}^{C-1} y_{nc}\log \hat{y}_{nc} $$
As I understand, $\hat{y}_{nc}$ is the probability that example $n$ belongs to class $c$ and $y_{nc}$ equals 1 when example $n$ belongs to class $c$ since it is a one-hot encoding. So does that mean there is a discrete random variable with support $\{0,1,\dots,9\}$ that takes on the probabilities $\hat{y}$? But isn't the ground truth deterministic, so is the true probability distribution that $y$ is part of even a probability distribution if there is no randomness in $y$? How would this be written using probability distributions?
 A: For two probability mass functions $p\left(x\right) = \operatorname{Pr}_{p}\left(X = x\right)$ and $q\left(x\right) = \operatorname{Pr}_{q}\left(X = x\right)$ of a random variable $X$ on the same support $\mathcal{X}$ (where $q\left(x\right)$ may be viewed as the 'approximating distribution' for $p\left(x\right)$), the cross-entropy between $p$ and $q$ is defined as
$$
\begin{align}
\operatorname{H}_{p, q} &= -\mathbb{E}_{p}\left[\log q\left(X\right)\right]\\
&= -\sum_{x_{i} \in \mathcal{X}}p\left(x_{i}\right)\log q\left(x_{i}\right)
\end{align} $$
This definition can also be extended to multivariate distributions. In particular, the joint cross-entropy involving the $N$-variate distributions of mutually independent random variables is just the sum of the corresponding marginal cross-entropies, so it can be written like:
$$\operatorname{H}_{p, q} = \sum_{n = 0}^{N - 1}\operatorname{H}_{p_{n}, q_{n}}$$
In machine learning, we typically treat the empirical distribution of the training data as the distribution we wish to approximate using some parametrised model class. The output vectors in a multi-class classification problem are usually represented as a 'one-hot' vector, eg. $y_{n} = \left(y_{n1}, y_{n2}, y_{n3}\right) = \left(0, 1, 0\right)$. If we wanted to fit our model to a single training example $\left(y_{n}, x_{n}\right)$ by minimising cross-entropy, then our objective function becomes
$$ J_{n}\left(\theta\right) = -\sum_{c = 0}^{C - 1}y_{nc}\log \hat{y}_{nc}\left(x_{n}; \theta\right)$$
where $\hat{y}_{nc}\left(x_{n}; \theta\right)$ denotes the model's predicted probability $x_{n}$ belonging to class $c$, parametrised in $\theta$.
Now if our training data consists of $N$ examples, then to minimise the joint cross-entropy from the empirical distribution of the training data, we can apply a standard independence assumption and take the objective function to be the sum of the cross-entropies for each individual example (a factor of $1/N$ can be included, but this does not affect the overall solution of the optimisation problem).
$$ J\left(\theta\right) = -\dfrac{1}{N}\sum_{n = 0}^{N - 1}\sum_{c = 0}^{C - 1}y_{nc}\log \hat{y}_{nc}\left(x_{n}; \theta\right)$$
