First of all, not specific to neural networks, you can think almost everything in the context of probabilistic frameworks. Given the data, call $x$, the label, or value in regression problems, call $y$, of each data point is unknown and can be thought of as random variable, in which you actually want to model the posterior, i.e. $P(y|x)$, to decide the label/value of the data point. Specifically in softmax, or logistic regression the outputs you get can be interpreted as the posterior distribution.
Even your data, $x$, represent samples drawn from a distribution, so may be treated as random variable. For example, think about collecting survey data via street interviews; the people answering your questions have some element of randomness.
Your weights (now being a little specific to neural networks) can also be interpreted in a probabilistic setting, by just considering them as random variables, and assigning some prior distribution to them. The prior distribution on weights corresponds to applying some sort of regularization in the optimization sense.
It might be useful for you to read through Bayesian Linear Regression to quickly grasp a very similar but simplified idea.