How Probability distribution relates to neural networks?

The concept of random variables and probability distributions are confusing in the context of neural networks.

In a neural network, which is the random variable and what is the probability distribution of it?

How does the output probability of a (say softmax output layer) affects or relates to the random variable?

How is this random variable and probability distribution thing related to neural networks or in general machine learning?

First of all, not specific to neural networks, you can think almost everything in the context of probabilistic frameworks. Given the data, call $$x$$, the label, or value in regression problems, call $$y$$, of each data point is unknown and can be thought of as random variable, in which you actually want to model the posterior, i.e. $$P(y|x)$$, to decide the label/value of the data point. Specifically in softmax, or logistic regression the outputs you get can be interpreted as the posterior distribution.
Even your data, $$x$$, represent samples drawn from a distribution, so may be treated as random variable. For example, think about collecting survey data via street interviews; the people answering your questions have some element of randomness.