In Goodfellow, Bengio, and Courville's book Deep Learning they state some variation of the following at several points (188.8.131.52, 10.2.3):
In general, we can think of the neural network as representing a function $f(x; \theta)$. The outputs of this function are not direct predictions of the value $y$. instead, $f(x; \theta)$ = $\omega$ provides the parameters for a distribution over $y$. Our loss function can then be interpreted as $-\log p(y;\omega(x))$. [p. 182]
I take this to mean that although a basic, standard neural network (i.e. feedforward with one output unit) outputs a single value, the training process and the suggested use of the maximum likelihood principle mean that inference is equivalent to sampling from some probability distribution whose structure is specified by the input and the network's parameters.
Am I interpreting this correctly? If so, what exactly is the relationship between the probability distribution and the network or the network's parameters? What is the relationship between $\omega$ and the network or the network's parameters?
Or are the distribution and parameters $\omega$ just hypotheticals and not meant to be interpreted so literally?