All Questions
Tagged with neural-networks probability
77 questions
2
votes
2
answers
37
views
Predicting the probability distribution of a deterministic dataset
In classical machine learning regression, we often assume the target variable $y$, given an input $x$, follows a probability distribution, allowing us to model and predict not just the expected value ...
- 121
1
vote
1
answer
38
views
Understanding deep learning notation and properties of expectation - neighbor2neighbor
I am trying to follow the proof of Theorem 1 here but can't fully understand the notation the authors use so that I am not able to completely understand the steps. This misunderstanding keeps coming ...
- 125
3
votes
1
answer
81
views
What probability distribution is learned in this specific case? [duplicate]
I keep reading papers and blogposts where the training of a neural network is defined as learning some underlying probability distribution of the data.
Imagine that you write CNN that outputs whether ...
- 173
2
votes
0
answers
173
views
Maximum Mean Discrepancy (MMD) implementation as a metric to measure GAN performance [closed]
I am trying to evaluate the performance of the GAN model, I have trained. I found that there exist two major choices FID (Fréchet inception distance) and MMD (Maximum Mean Discrepancy) for comparing ...
- 21
2
votes
1
answer
236
views
Error rate vs Empirical risk - What's the difference between practical and theoretical terms for performance of neural networks?
Motivation
I am currently reading the following book: Understanding machine learning by Shalev-Shwartz and Ben-David. The book uses statistics terminology in its machine learning theory, and it is not ...
- 21
4
votes
4
answers
171
views
Why do we work with factor of likelihoods instead of e.g. a sum for a batch in the negative log likelihood loss function?
In a classification task, at a certain stage of the training process, we get a likelihood of sampling proper class Y for a particular data point X. For batch, we get many independent likelihoods.
Let'...
2
votes
0
answers
72
views
How does Dempster-Shafer Theory of Evidence relate to Deep Learning?
I am reading this article and it has the following phrase - "Dempster-Shafer Theory of Evidence assigns belief masses a set of classes (unlike assigning a probability to a single class)". ...
- 644
6
votes
3
answers
2k
views
Have you ever seen anyone mention probability density function as a framework for neural network?
I am evaluating a proposal pitch from a vendor about their machine learning solution. I do not have access to the source code or any other technical details about the algorithm they are using as it is ...
- 548
1
vote
0
answers
215
views
Softmax Response vs MC Dropout for Uncertainty Estimation [closed]
Some papers I see take the uncertainty estimation of a prediction as simply its softmax/sigmoid output, whereas some papers will use techniques such as MC Dropout and calculate the variance across the ...
- 11
2
votes
0
answers
118
views
How to prove that neural network estimates posterior distribution
Let's say that I train a neural network in a classic binary classification setting where all the training data has labels in $\{-1, +1\}$. From my understanding, if I train the network with a log-loss ...
- 53
0
votes
1
answer
102
views
Normalizing Flows Invertibility
I am currently reading up on RealNVP, which has the following transformations according Lilian Weng:
$$
\begin{aligned}
\mathbf{y}_{1:d} &= \mathbf{x}_{1:d} \\
\mathbf{y}_{d+1:D} &= \mathbf{x}...
- 19
7
votes
1
answer
939
views
Why must a product of symmetric random variables be symmetric?
I was reading about weight initialization in neural networks (He et. al, 2015) when I came across this statement:
"If we let $w_{l-1}$ have a symmetric distribution around zero and $b_{l} = 0$, ...
- 173
1
vote
0
answers
38
views
Neural Networks Miscalibration Measure
I have read these two papers related to the neural network miscalibration problem: "On Calibration of Modern Neural Networks" and "Multivariate Confidence Calibration for Object ...
- 11
1
vote
0
answers
17
views
When imputing missing labels Y1 == NaN during training, how do additional target vectors (Y2 != NaN) impact learning Y1==NaN?
I am training a Mixture Density Network (MDN) to map from continuous input vectors X to continuous targets Y [i.e. X -> Y]. There are missing labels on vector Yi, which I impute from the mdn (as ...
- 11
0
votes
0
answers
87
views
Normalizing Flow Penalization
I am looking to train a normalizing flow, specifically a Masked Autoregressive Flow model. However, this model leads to high variance on lower dimensional, less complex data. I am using a neural ...
- 19
3
votes
1
answer
202
views
MNIST with a TWIST, no labels given, only probabilities
Let's say we have basic MNIST dataset, and we have the same goal to predict the digit, BUT we're swapping all the labels by RED ...
- 141
0
votes
3
answers
648
views
Is Regression and Classification "Inherently" Based on Probability?
From a classical perspective, I have outlined some examples of models in which Probability seems to play an "inherent role" in Regression and Classification:
As a simple example, suppose we ...
1
vote
0
answers
165
views
How to understand the density in machine learning?
We can calculate the conditional density using Eq.1[3].
$$
p_{\theta, \Lambda}(y \mid \boldsymbol{x})=\frac{\exp \left(f_{\theta, \Lambda}(\boldsymbol{x})[y]\right)}{\sum_{k=1}^{n} \exp \left(f_{\...
- 11
1
vote
0
answers
258
views
How to sample from a distribution approximated by a Neural Network?
There are a few models already that approximate distributions with a neural network i.e.: energy models define a density function $f(x)= e^{S(x,w)}/Z$ where $S$ is a neural network and $Z$ is a ...
- 768
2
votes
2
answers
544
views
Does Multinomial Probability Calibration Consider the Probabilities of the Non-Dominant Classes?
The gist behind Harrell's rms::calibrate function makes sense to me. While I have yet to understand the magic that lets us calculate the "true" ...
- 67.1k
1
vote
1
answer
1k
views
Converting weighted value to probability
For a disease classification problem I have trained a deep learning model to predict the probability score of the disease based on images only. During training time I had access to labeled images only ...
- 13
7
votes
3
answers
2k
views
How much of neural network overconfidence in predictions can be attributed to modelers optimizing threshold-based metrics?
Neural network "classifiers" output probability scores, and when they are optimized via crossentropy loss (common) or another proper scoring rule, they are optimized in expectation by the ...
- 67.1k
3
votes
1
answer
80
views
Deriving bayes formulas from "Overcoming catastrophic forgetting in neural networks"
I am trying to understand the formulas from the paper "Overcoming catastrophic forgetting in neural networks" and am wondering if someone could help explain how they derive these formulas. ...
- 75
1
vote
0
answers
66
views
Skipgram model theory confusion
In the output layer of a skipgram model, there are $|\text{Context}|*|\text{Vocab}|$ values. And for each context word, the values are basically the dot product of the input word representation and ...
- 133
1
vote
0
answers
264
views
Softmax Classifier gives weird confusion matrix
I'm currently working on a problem of binary classification in keras and have decided to use the softmax function as the activation function for my final classification layer. My current network is as ...
3
votes
1
answer
1k
views
How to find bits/dim of a gaussian output distribution?
I have images that are 64x64x3 and 64x64x1 8-bit. I transform those images down to [-1,1]. I now want to find the bits/dim for my VAE log probability. How do I find the bits/dim of the log likelihood? ...
- 65
1
vote
1
answer
127
views
Undergrad sources for information theory
If this is out of topic and going to be closed, I will appreciate to know where it is right to ask this question, as I am kind of lost right now.
I am a software engineering BSc, and recently started ...
1
vote
1
answer
598
views
how does the posterior predictive distribution account for differences in uncertainty?
Recent research in the field of bayesian deep learning allows for the quantification of uncertainty in estimates of ML models. This can be done because the posterior predictive distribution of the ...
- 1,116
1
vote
0
answers
642
views
KL divergence of Variational Autoencoder not decreasing
I have been trying to train a VAE to generate cat pictures. The images are of size 64 by 64. But upon training my reconstruction loss decreases whereas my KL loss remains constant/slightly increases. ...
- 11
3
votes
0
answers
122
views
Correlated random variables and ensembles (law of large numbers?)
Consider $n$ i.i.d random variables. By the law of large numbers (LLN) the sample average would converge after some time to the expected value.
Let's assume the random variables are correlated. Would ...
- 538
0
votes
1
answer
435
views
About Murphy's notation: why is $p(y|x, \theta)$ a conditional expectation when there is no probabilistic interpretation on $x$ or $\theta$?
In section 1.4.5 of Kevin Murphy ML textbook, he introduces linear regression where for a given data $x$, the target $y$ is assumed to be obtained through
$$y(x) = w^Tx + \epsilon, \text{ where } \...
- 357
0
votes
0
answers
62
views
Understanding deterministic models via probabilistic graphical model
I have read a few tutorials how we can think of deterministic neural networks with the help of probabilistic graphical models. Very often they would offer an image as seen bellow and say, our model is ...
- 261
2
votes
1
answer
3k
views
Adding random noise to latent representation increase the accuracy in the autoencoder
I am working on an autoencoder project, it consists of dense layers like this :
...
- 333
1
vote
1
answer
548
views
Probability that a population mean exceeds a threshold
In Googling this question, I see that there are a variety of similar tests but I couldn't find anything given the exact way I'm approaching this problem. This might be something obvious but I'm not ...
- 173
2
votes
1
answer
511
views
Is there a typo in the paper "Evolution Strategies as a Scalable Alternative to reinforcement learning" by OpenAI?
Original paper: https://arxiv.org/pdf/1703.03864.pdf
On page 2-3, it writes,
But this equation is clearly wrong.
Since the step needed to derive the equation involves the argument, let $$\theta = \...
- 734
1
vote
1
answer
699
views
What is probability distribution for a machine learning task?
I am new to machine learning. I feel confused of how to understand the probability distribution of the training set, like $p(y|x)$ and $p(x)$, where $x$ is a training sample and $y$ is a label. Is ...
- 155
0
votes
0
answers
26
views
How to name a machine learning algorithm where we used very small amount of labeled data?
I have to describe a machine-learning algorithm that needs only very small dataset. I cannot say it is unsupervised because I already used the labels in the training. Also, I cannot say semi-...
- 159
1
vote
1
answer
60
views
Can anyone help to explain one of the variables in a figure that illustrates how posterior probabilities shift and move around?
I am learning this post.
The book gives this figure to illustrate how posterior probabilities shift and move around
Here is the code
...
- 566
5
votes
1
answer
2k
views
KL divergence of a uniform prior and a custom posterior
So I was reading the Google's paper on VQ-VAE and have stumbled upon the derivation of KL divergence of the uniform prior and the given distribution:
$$q(z=k \mid x)=\left\{\begin{array}{ll}{1} & ...
- 75
2
votes
2
answers
28
views
Theoretical grounding for ease of training with a prior
If we have a neural network that learns the generative model for P(A, B, C)
And now, we want to learn the generative model for P(A, B, C, D)
Is there any theory that says learning P(A,B,C) and then ...
- 621
7
votes
4
answers
4k
views
Model ensembling - averaging of probabilities
From the BatchNorm paper, section 4.2.3, (https://arxiv.org/abs/1502.03167),
The ensemble prediction was based on the arithmetic average of class
probabilities predicted by the constituent ...
- 121
3
votes
1
answer
916
views
How is the standard deviation of VAE's constructed?
I am trying to build a Variational Autoencoder. I was looking at various codes online and found most of them in some way or another copy Francois Chollet (Google researchers) code.
Now my main ...
- 215
0
votes
2
answers
649
views
How many neurons are actually dropped when using dropout?
I understand that when using dropout, a single neuron can be described using Bernoulli random variable and for a set of neurons it can be described as Binomial random variable
When using Dropout, we ...
- 195
5
votes
1
answer
118
views
Convergence to gradient in limit of variance
I came across this equation in the original GAN paper (pg 2 https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf):
$$\lim_{\sigma \rightarrow 0}
\nabla_{\bf x} \mathbb{E}_{\epsilon \sim \...
- 205
1
vote
0
answers
414
views
Did I understand the usage of Gumbel-Softmax reparametrization correctly?
I am working on a deep learning model, which has a mixture of experts formulation like $\log p(y|x)=\log \sum_{z}p(y|z,x,\theta)p(z|x,\phi)$. So, each $p(y|z,x,\theta)$ is a deep learning classifier, ...
- 2,310
0
votes
0
answers
108
views
Joint probability distribution of correlated data points
I have a query with respect to joint distributions.
Here, each output data point in $\mathbf{y}$ is conditionally independent given the inputs $\mathbf{x}$ and the mapping $f:\mathbf{x}\rightarrow \...
- 31
1
vote
1
answer
67
views
Bayesian statistics: probability of next point
I am reading the Deep Learning book and having some difficulties with the following formula (page 134):
$$
p(X^{m+1} | x^1, \dots, x^m) = \int p(X^{m+1} | \theta) p(\theta | x^1, \dots, x^m) d\theta.
...
- 729
1
vote
1
answer
572
views
What are some good canned classifiers for high-dimensional data with probablistic labels, besides neural nets?
I've got a classification problem where my labels are $N\times4$ matrices of probabilities of class membership, and I've got about 1800 covariates. The covariates are mostly granular, in the sense ...
- 13.7k
13
votes
3
answers
14k
views
Variational autoencoder: Why reconstruction term is same to square loss?
In variational autoencoder (see paper), page 5, the loss function for neural networks is defined as:
$L(\theta;\phi;x^{i})\backsimeq 0.5*\sum_{j=1}^J(1 + 2\log\sigma^i_j-(\mu^i)^2) - (\sigma^i)^2) + \...
- 420
1
vote
0
answers
35
views
Normalization: different approach
Today I had a discussion about the right way to normalize data, especially image data. The standard approach, as found in many tutorials etc., seems to be following. For example, the data has a range ...
- 11