0
$\begingroup$

When we use deep neural networks (DNNs) to solve a 1-dimention regression problem, we can approximate data distribution with the output of a DNN like the picture below.
My question is that DNN does not have the assumption of gaussian distribution or any other distribution of itself. It just knows what value to output when it sees an input. So how do you know the probability distribution of the DNN? For example, if someone asks, what is the probability of the point appearing in (5, 0). Can DNN answer this kind of questions?

enter image description here (pic from https://medium.com/@sunnerli/dnn-regression-in-tensorflow-16cc22cdd577)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

For many regression algorithms, not only neural networks, the model is that the data is distributed by $y \sim \mathcal{N}(f(x;\theta), \sigma^2)$, where $\theta$ are the model parameters and $\sigma^2$ is the variance of the distribution (often a hyperparameter).

Maximizing the log-likelihood of the data with respect to $\theta$ is equivalent to minimizing the mean squared error loss between the $y_i$ and $f(x_i;\theta)$.

Therefore, to compute the probability density of $(5,0)$, you would just find the density of a gaussian with mean $f(5; \theta)$ and a variance of $\sigma^2$, where $f$ is your neural network.

Improve this answer
$\endgroup$
13
  • $\begingroup$ Thanks for your answer. But I still have two questions. 1: Does apply DNN to a regression problem also must have the assumption that the data has to be gaussion distributed? As I know, we just care about mean square error (MSE) of difference of output value and ground truth value. There is no gaussian distribution involved. 2. How do I find out a DNN's multiple means and variances from its weights and biases only? Is this even possible? $\endgroup$
    – Lion Lai
    Jan 30, 2018 at 4:09
  • $\begingroup$ 1. Using MSE (L2) loss corresponds to the data being distributed normally. Using L1 loss corresponds to data being distributed according to the laplacian distribution. In general, there is a mapping between loss functions and probability distributions. 2. Not sure what you mean by a DNN's multiple means and variances. $\endgroup$
    – shimao
    Jan 30, 2018 at 4:11
  • $\begingroup$ 1. Are you refering to regularization? Can you add references of them? 2. After traing a DNN model, all we have are network's weights and biases. How can I calculate the density funtion from these numbers. Thank you. $\endgroup$
    – Lion Lai
    Jan 30, 2018 at 4:18
  • 1
    $\begingroup$ If you train an DNN like usual, you can already interpret the ouput as a probability distribution. It is a gaussian distribution. The mean of the gaussian distribution is the output of the network, and the variance of the distribution is a fixed hyperparameter. There is no additional step that you have to do. $\endgroup$
    – shimao
    Jan 31, 2018 at 4:14
  • 1
    $\begingroup$ Yes, it still applies. The only assumption we need to apply MSE loss and obtain our PDF is that the distribution of the target $y$ is a function of the input $x$ plus some gaussian noise. That function $f$ doesn't have to be linear and in this case is modeled by a highly nonlinear neural network. $\endgroup$
    – shimao
    Feb 2, 2018 at 2:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.