All Questions
40 questions
2
votes
2
answers
37
views
Predicting the probability distribution of a deterministic dataset
In classical machine learning regression, we often assume the target variable $y$, given an input $x$, follows a probability distribution, allowing us to model and predict not just the expected value ...
- 121
3
votes
1
answer
81
views
What probability distribution is learned in this specific case? [duplicate]
I keep reading papers and blogposts where the training of a neural network is defined as learning some underlying probability distribution of the data.
Imagine that you write CNN that outputs whether ...
- 173
2
votes
0
answers
174
views
Maximum Mean Discrepancy (MMD) implementation as a metric to measure GAN performance [closed]
I am trying to evaluate the performance of the GAN model, I have trained. I found that there exist two major choices FID (Fréchet inception distance) and MMD (Maximum Mean Discrepancy) for comparing ...
- 21
4
votes
4
answers
171
views
Why do we work with factor of likelihoods instead of e.g. a sum for a batch in the negative log likelihood loss function?
In a classification task, at a certain stage of the training process, we get a likelihood of sampling proper class Y for a particular data point X. For batch, we get many independent likelihoods.
Let'...
2
votes
0
answers
72
views
How does Dempster-Shafer Theory of Evidence relate to Deep Learning?
I am reading this article and it has the following phrase - "Dempster-Shafer Theory of Evidence assigns belief masses a set of classes (unlike assigning a probability to a single class)". ...
- 644
6
votes
3
answers
2k
views
Have you ever seen anyone mention probability density function as a framework for neural network?
I am evaluating a proposal pitch from a vendor about their machine learning solution. I do not have access to the source code or any other technical details about the algorithm they are using as it is ...
- 548
1
vote
0
answers
38
views
Neural Networks Miscalibration Measure
I have read these two papers related to the neural network miscalibration problem: "On Calibration of Modern Neural Networks" and "Multivariate Confidence Calibration for Object ...
- 11
0
votes
0
answers
87
views
Normalizing Flow Penalization
I am looking to train a normalizing flow, specifically a Masked Autoregressive Flow model. However, this model leads to high variance on lower dimensional, less complex data. I am using a neural ...
- 19
3
votes
1
answer
202
views
MNIST with a TWIST, no labels given, only probabilities
Let's say we have basic MNIST dataset, and we have the same goal to predict the digit, BUT we're swapping all the labels by RED ...
- 141
0
votes
3
answers
648
views
Is Regression and Classification "Inherently" Based on Probability?
From a classical perspective, I have outlined some examples of models in which Probability seems to play an "inherent role" in Regression and Classification:
As a simple example, suppose we ...
1
vote
0
answers
165
views
How to understand the density in machine learning?
We can calculate the conditional density using Eq.1[3].
$$
p_{\theta, \Lambda}(y \mid \boldsymbol{x})=\frac{\exp \left(f_{\theta, \Lambda}(\boldsymbol{x})[y]\right)}{\sum_{k=1}^{n} \exp \left(f_{\...
- 11
1
vote
0
answers
258
views
How to sample from a distribution approximated by a Neural Network?
There are a few models already that approximate distributions with a neural network i.e.: energy models define a density function $f(x)= e^{S(x,w)}/Z$ where $S$ is a neural network and $Z$ is a ...
- 768
2
votes
2
answers
544
views
Does Multinomial Probability Calibration Consider the Probabilities of the Non-Dominant Classes?
The gist behind Harrell's rms::calibrate function makes sense to me. While I have yet to understand the magic that lets us calculate the "true" ...
- 67.1k
7
votes
3
answers
2k
views
How much of neural network overconfidence in predictions can be attributed to modelers optimizing threshold-based metrics?
Neural network "classifiers" output probability scores, and when they are optimized via crossentropy loss (common) or another proper scoring rule, they are optimized in expectation by the ...
- 67.1k
1
vote
0
answers
264
views
Softmax Classifier gives weird confusion matrix
I'm currently working on a problem of binary classification in keras and have decided to use the softmax function as the activation function for my final classification layer. My current network is as ...
3
votes
1
answer
1k
views
How to find bits/dim of a gaussian output distribution?
I have images that are 64x64x3 and 64x64x1 8-bit. I transform those images down to [-1,1]. I now want to find the bits/dim for my VAE log probability. How do I find the bits/dim of the log likelihood? ...
- 65
1
vote
1
answer
127
views
Undergrad sources for information theory
If this is out of topic and going to be closed, I will appreciate to know where it is right to ask this question, as I am kind of lost right now.
I am a software engineering BSc, and recently started ...
3
votes
0
answers
122
views
Correlated random variables and ensembles (law of large numbers?)
Consider $n$ i.i.d random variables. By the law of large numbers (LLN) the sample average would converge after some time to the expected value.
Let's assume the random variables are correlated. Would ...
- 538
0
votes
1
answer
435
views
About Murphy's notation: why is $p(y|x, \theta)$ a conditional expectation when there is no probabilistic interpretation on $x$ or $\theta$?
In section 1.4.5 of Kevin Murphy ML textbook, he introduces linear regression where for a given data $x$, the target $y$ is assumed to be obtained through
$$y(x) = w^Tx + \epsilon, \text{ where } \...
- 357
2
votes
1
answer
3k
views
Adding random noise to latent representation increase the accuracy in the autoencoder
I am working on an autoencoder project, it consists of dense layers like this :
...
- 333
2
votes
1
answer
511
views
Is there a typo in the paper "Evolution Strategies as a Scalable Alternative to reinforcement learning" by OpenAI?
Original paper: https://arxiv.org/pdf/1703.03864.pdf
On page 2-3, it writes,
But this equation is clearly wrong.
Since the step needed to derive the equation involves the argument, let $$\theta = \...
- 734
1
vote
1
answer
699
views
What is probability distribution for a machine learning task?
I am new to machine learning. I feel confused of how to understand the probability distribution of the training set, like $p(y|x)$ and $p(x)$, where $x$ is a training sample and $y$ is a label. Is ...
- 155
0
votes
0
answers
26
views
How to name a machine learning algorithm where we used very small amount of labeled data?
I have to describe a machine-learning algorithm that needs only very small dataset. I cannot say it is unsupervised because I already used the labels in the training. Also, I cannot say semi-...
- 159
1
vote
1
answer
60
views
Can anyone help to explain one of the variables in a figure that illustrates how posterior probabilities shift and move around?
I am learning this post.
The book gives this figure to illustrate how posterior probabilities shift and move around
Here is the code
...
- 566
5
votes
1
answer
2k
views
KL divergence of a uniform prior and a custom posterior
So I was reading the Google's paper on VQ-VAE and have stumbled upon the derivation of KL divergence of the uniform prior and the given distribution:
$$q(z=k \mid x)=\left\{\begin{array}{ll}{1} & ...
- 75
5
votes
1
answer
118
views
Convergence to gradient in limit of variance
I came across this equation in the original GAN paper (pg 2 https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf):
$$\lim_{\sigma \rightarrow 0}
\nabla_{\bf x} \mathbb{E}_{\epsilon \sim \...
- 205
0
votes
0
answers
108
views
Joint probability distribution of correlated data points
I have a query with respect to joint distributions.
Here, each output data point in $\mathbf{y}$ is conditionally independent given the inputs $\mathbf{x}$ and the mapping $f:\mathbf{x}\rightarrow \...
- 31
1
vote
1
answer
572
views
What are some good canned classifiers for high-dimensional data with probablistic labels, besides neural nets?
I've got a classification problem where my labels are $N\times4$ matrices of probabilities of class membership, and I've got about 1800 covariates. The covariates are mostly granular, in the sense ...
- 13.7k
1
vote
2
answers
354
views
Probabilistic model of neural network
I read Neil's presentation and found this joint model is confusing:
$$p(y_*|y, X, x_*) = \int p(y_*|x_*, W)p(W|y, X)dW $$
where $W$ contains $W_1$ and $W_2$ and $p(W|y,X)$ is posterior density
Can ...
- 547
1
vote
0
answers
704
views
How can an ensemble perform worse than all but one of its constituents?
I came across a very unusual situation:
I trained 5 deep nets on a problem.
4 of the 5 had excellent in- and out-of-sample accuracy.
I trained a classifier on the probability outputs of the 5 deep ...
- 743
5
votes
2
answers
2k
views
Is graduate level probability theory (Durett) used often in ML, DL research?
I am interested in machine learning; I have a particular liking for RNNs. I have coursework in some areas of computer science, e.g., data mining, optimization for ML algorithms, deep learning, and an ...
- 5,800
4
votes
2
answers
8k
views
Softmax Multiclass Classification
How do we associate a class to every output unit in a multilayer neural network architecture? I mean we assign the output to the class with maximum probability, but how do we decide which neuron ...
- 211
10
votes
4
answers
25k
views
Neural networks output probability estimates?
Suppose my training data contains ~100 variables, and each example is tagged as "success" or "failure".
I understand how a neural network can be used to try and predict success vs failure based on ...
- 219
8
votes
1
answer
9k
views
The effect of temperature in temperature sampling
I was reading this while I found:
The high temperature sample displays greater linguistic variety, but
the low temperature sample is more grammatically correct. Such is the
world of temperature ...
- 1,395
-1
votes
1
answer
123
views
Regularity of functions approximated with neural networks
Are there any papers pertaining to smoothness / regularity of functions that are approximated with artificial neural network?
0
votes
2
answers
2k
views
Getting probability from Restricted Boltzmann Machine
Let's consider a trained Restricted Boltzmann Machine model. It was trained to maximize P(v). Since it's a generative model, how can I get a probability of an input vector which it is supposed to ...
- 21
4
votes
2
answers
1k
views
How to evaluate the quality of the probability distribution output of a classifier?
In a classification problem, I have trained a neural network which outputs class probabilities for a given input. For a new input, I now want to evaluate the "quality" of the neural network's ...
- 7,129
7
votes
1
answer
3k
views
Combine several softmax output probabilities
I would like to combine the outputs of five neural networks, each with a softmax output layer of three classes each. A typical, example output is shown below:-
where Figure 1 is the output of model 1,...
- 4,869
7
votes
4
answers
8k
views
Methods & CRAN packages to predict probability using neural networks or others machine learning algorithms
I have a medical database containing 7 input variables (4 are binary) and a binary outcome variable (Survival: yes/no). My objective is to train and test an algorithm that predict probability of ...
- 75