33
votes
Why is softmax output not a good uncertainty measure for Deep Learning models?
This question can be answered more precisely than the current answers. Fixing the deviation between the predicted probabilities (the output of the softmax layer of a neural network) and their true ...
- 449
16
votes
Accepted
Laymen Statistics Talk
There are several statistical issues that are relevant to this short dialog.
The fact that John spilled his glass of wine on the table at that exact moment is peculiar. Never have I seen a man so ...
- 20.6k
16
votes
Accepted
Finding most likely permutation
Provided the measurement errors are independent and identically Normally distributed for each instrument, the solution is to match the two sets of measurements in sorted order. Although this is ...
- 334k
15
votes
Accepted
Why is it called standard “error” and not standard “uncertainty”?
The question "why is this term used, rather than this other term" is, like much terminology, a matter of historical happenstance. Sometimes the outcome is felicitous and sometimes less so. I ...
- 291k
12
votes
Accepted
How does the Brier Score break down to (Reliability - Resolution + Uncertainty)?
Brier Score decomposition relies on a foundational concept in statistics called the partition of sums of squares. The basic idea is that we have a sums of squares over any series, we can break that ...
- 3,335
11
votes
Why is softmax output not a good uncertainty measure for Deep Learning models?
The relationship between softmax confidence and uncertainty is more
complicated than a lot of work makes it sound.
Firstly, there are two separate issues that often get conflated.
Callibration - Does ...
- 746
9
votes
Accepted
How would one find the uncertainty in a mean if the data points themselves have zero-order uncertainty?
Assuming the observations are collected independently of each other, the easiest way I can think of is to propagate uncertainty by using simulation. The idea is to generate random vectors from the (...
- 12.1k
8
votes
Reliability of a fitted curve?
This is an ordinary least squares problem!
Defining
$$x = V^{-2/3}, \ w = V_0^{1/3},$$
the model can be rewritten
$$\mathbb{E}(E|V) = \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x^3$$
where the ...
- 334k
8
votes
Accepted
Does it make sense to find confidence intervals for neural networks?
For simple regression models, if you have the joint distribution of the parameters you get both confidence intervals and in a sort of derived fashion prediction intervals. You typically have the joint ...
- 35.2k
7
votes
Accepted
Working out error on fit parameters for nonlinear fit
The $f_i$ is the residual of the model fit for the $i$th data point.
$f_i = y_i - a * \tan^{-1}(b x_i)$
$J^TJ$ may be inaccurate estimate of Hessian:
Note that the Hessian equals $J^TJ$ plus a (...
- 13.5k
7
votes
Accepted
How to estimate the uncertainty in the zeros of a fitted function?
The principal objective of this reply is to point out how perilous this enterprise can be. Along the way I'll be able to suggest some approaches as well as provide some ideas for a different analysis....
- 334k
7
votes
Uncertainty propagation for the solution of an integral equation
Let's break this down into easier problems. To keep the post reasonably short, I will only sketch a good confidence interval procedure without going into all the details.
What is interesting about ...
- 334k
6
votes
Accepted
Two measurement devices vs 1 device multiple measurements
Regardless of how these devices behave, an additive model of variability provides useful insight.
Such a model supposes that the response of an instrument is the sum of three independent quantities (...
- 334k
6
votes
Bootstrap intervals for predictions, how to interpret it?
Yes, it is perfectly sensible.
For a quick interpretation, I like the one provided by Davison: Assuming $T$ is an estimator of a parameter $\psi$ based on a random sample $Y_1, . . . , Y_n$, $V_T^{0.5}...
- 46k
6
votes
Accepted
theoretical confidence interval depending on sample size
Nice experiment. The blue lines will be at $\mu \pm z_{\alpha/2} \sigma/\sqrt{n}$ where $\alpha = 0.05$ and $\alpha \mapsto z_\alpha$ is the upper quantile function of the standard normal and $n$ is <...
- 1,841
5
votes
Detection Fraction from Counting Experiment: Poisson or Binomial Uncertainty?
The two seem to be based on different experimental setups, or at least in terms of what parameters are considered known vs. uncertain.
Your Binomial approach assumes the number of objects $N=100$ is ...
- 13.1k
5
votes
Why is softmax output not a good uncertainty measure for Deep Learning models?
Softmax distributes the 'probability' 0-1 between the available classes.
It does not express incertitude, it is not a PDF function.
If you want to express the incertitude you should be looking into
...
- 51
5
votes
Why, *intuitively*, in regular parametric problems, does uncertainty go down at a $\sqrt{ n }$ rate on the SE/posterior SD scale?
The intuition for the variance is best understood in geometric terms, by likening the variance to a geometric analogy. With this analogy, the intuition of the additive nature of the variance operator ...
- 133k
5
votes
Is a statistically significant difference within analytical uncertainty still valid?
Suppose you are trying to weigh a package on a scale that gives unbiased readings but is subject to variations from one weighing to the next.
If the true weight of the box is 960g and we have the ...
- 57.6k
5
votes
What does "Aleatoric and Epistemic uncertainties" mean?
A short and very simplified literal explanation:
Aleatoric: uncertainty about the result of an experiment that we can repeat, e.g. dice roll. What is the probability of rolling a 6? - the view of ...
- 301
5
votes
How many significant figures should I report for a regression equation?
My eyebrows go up when numbers are reported with far too much precision, too, but there's more going on in a regression setting than we might expect. So much, so, that I won't venture a thorough ...
- 334k
5
votes
Finding most likely permutation
@whuber (+1) has answered the question in your title about finding the most likely permutation. My purpose here is to explore briefly by simulation whether you can expect that most likely permutation ...
- 57.6k
5
votes
Accepted
What is the difference between a non-zero nugget and a noise term in Kriging/GPR?
Random noise and nugget effect are indeed quite similar to some extent. The difference between the two appears
when there are repeated observations (i.e., several observations at the same location), ...
- 103
5
votes
Accepted
Errors on cubic fits are relative to the order of magnitude of the y-data, not fit quality
The main problem is that the measure of "error on the intercepts" used in the question is a percentage error, of the standard error relative to the intercept. That's a problem with a least-...
- 102k
4
votes
Visualising uncertainty in slope and offset for a regression line?
A sometimes useful alternative to the confidence band is to show the estimated line together with bootstrapped lines. Here is a simple, simulated example:
EDIT
...
- 82.8k
4
votes
Accepted
How to incorporate uncertainty of actual historical data into forecast prediction intervals?
No, there is no such function in forecast, nor in any other R package that I am aware of.
The conceptually simplest way of going about this would be to take your ...
- 131k
4
votes
Derivation of uncertainty propagation?
The idea behind the differential calculus is to study potentially complicated functions $f:\mathbb{R}^n \to \mathbb{R}^m$ by means of linear approximations. Everything flows from this single idea.
...
- 334k
4
votes
Predicting Uncertainty in Random Forest Regression
As far as I know, the uncertainty of the RF predictions can be estimated using several approaches, one of them is the quantile regression forests method(Meinshausen, 2006), which estimates the ...
- 41
4
votes
Accepted
Uncertainty estimation in high-dimensional inference problems without sampling?
First of all, I think your statistical model is wrong. I change your notation to one more familiar to statisticians, thus let
$$\mathbf{d}=\mathbf{y}=(y_1,\dots,y_N),\ N=10^6$$
be your vector of ...
- 18.4k
Only top scored, non community-wiki answers of a minimum length are eligible