1
$\begingroup$

The concept of random variables and probability distributions are confusing in the context of neural networks.

In a neural network, which is the random variable and what is the probability distribution of it?

How does the output probability of a (say softmax output layer) affects or relates to the random variable?

How is this random variable and probability distribution thing related to neural networks or in general machine learning?

$\endgroup$
1
$\begingroup$

First of all, not specific to neural networks, you can think almost everything in the context of probabilistic frameworks. Given the data, call $x$, the label, or value in regression problems, call $y$, of each data point is unknown and can be thought of as random variable, in which you actually want to model the posterior, i.e. $P(y|x)$, to decide the label/value of the data point. Specifically in softmax, or logistic regression the outputs you get can be interpreted as the posterior distribution.

Even your data, $x$, represent samples drawn from a distribution, so may be treated as random variable. For example, think about collecting survey data via street interviews; the people answering your questions have some element of randomness.

Your weights (now being a little specific to neural networks) can also be interpreted in a probabilistic setting, by just considering them as random variables, and assigning some prior distribution to them. The prior distribution on weights corresponds to applying some sort of regularization in the optimization sense.

It might be useful for you to read through Bayesian Linear Regression to quickly grasp a very similar but simplified idea.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.