All Questions
Tagged with uncertainty machine-learning
30 questions
4
votes
1
answer
119
views
Uncertainty of ANN outputs as distribution parameters
It is not an uncommon practice to train neural network models via negative log likelihood $-\mathcal{L}(x, y_{true}, \mu, \sigma)$ to estimate both a location ($\mu$) and a scale ($\sigma$), such that ...
- 167
0
votes
0
answers
15
views
How to determin the theoretical prediction limit for a complex process?
how can we find out what is the theoretical prediction limit for a complex process?
For example, for a coin toss (on average) the prediction limit is 50%, that is we cannot predict better than this ...
- 73
0
votes
0
answers
59
views
What can you do with quantified uncertainty in latent variable time-series models?
Uncertainty quantification in latent variable models is a topic I am interested in, but I am struggling to grasp the difference between what you can do with quantified parameter uncertainty and ...
- 1
1
vote
1
answer
476
views
Neural network regression - predicting mean and standard deviation
I have a dataset where for the same input, you get slightly varying results. In the final dataset I am using there are 4 input/output pairs for every input where the input is exactly the same but the ...
- 45
0
votes
1
answer
270
views
Evaluating the model stability using bootstrapping
I need help with the following.
Using our alternate data for external clients, we have built a model for identifying fraudulent customers (classification).
we used the auto-ml package to arrive at the ...
- 1
5
votes
1
answer
793
views
Aleatoric and Epistemic Uncertainty in the Framework of Bayesian and Frequentist
Beginning with definitions of Aleatoric and Epistemic Uncertainty from this paper:
Aleatoric: Aleatoric uncertainty refers to the intrinsic uncertainty of a particular system and the observed data. It ...
- 61
1
vote
0
answers
228
views
Probabilistic machine learning models: parameter uncertainty
Consider models such as DeepAR, ngboost and other frameworks to the general problem of predicting the parameters of some parametric distribution with some black-box function, call it f(X). The ...
- 831
1
vote
0
answers
71
views
Distributions as Features in Machine Learning
The Problem
Let's assume I have a problem that seems perfect for supervised learning. However, some of the measurements I would like to use as features are not point estimates but are instead ...
- 153
0
votes
0
answers
33
views
Uncertainty estimation in the input space
my input is an array between 0 and 1000 and the output is the corresponding system velocity. The input value is randomly generated (for instance by using the function in Python ...
- 101
0
votes
1
answer
60
views
Uncertainity band in Neural networks
I am working on a problem where I have to give the uncertainty band of my predictions like the image attached.
I have seen a StackExchange solution for this, but in the solution code, we are using ...
2
votes
1
answer
416
views
Aleatoric uncertainty in Gaussian Processes
I'm pretty new to statistics and machine learning and I'm studying and implementing GPs. One thing I observed in my studies is that in the figures representing all possible functions given the ...
1
vote
0
answers
21
views
Measuring uncertainty of model prediction by repeat measurements
Say I’ve trained some single value regression ML model (a neural network or something). I have trained this ML model with simulation data. I see that this neural network is good at predicting data in ...
- 111
1
vote
1
answer
280
views
How to calculate uncertainty for predictions coming from cascade of models?
I have developed a bunch of models to predict house prices. It is a 3 fold process:
I fit a gbm (first_model) and get the first prediction (first_pred),
there are some sub-models (simple lineer ...
- 219
0
votes
1
answer
46
views
Likelihood that a prediction falls above (below) 110% (90%) of the prediction
For my client I have to predict some products' prices with gbm (scikit). So in the production, I am to give prediction intervals. That is, I need to provide how likely a real price falls above 110% or ...
- 219
4
votes
2
answers
611
views
What does "Aleatoric and Epistemic uncertainties" mean?
While studying Bayesian Learning, I have encountered the term, Aleatoric and Epistemic uncertainty, but I have just found it a bit confusing to understand. I believe I haven't found good references to ...
- 3,585
1
vote
0
answers
154
views
How to calculate uncertainty from test set size
I'm training a machine learning model and trying to determine how large my test set should be. I'm not using any k-fold cross-validation, just the test set. I believe the only benefit of increasing ...
- 438
1
vote
0
answers
88
views
Propagating uncertainty through nested random forest models
Does anyone know if there are methods for propagating the prediction intervals (i.e. uncertainty) of nested surrogate models, specifically random forests? When I say nested, I mean that a second model ...
- 11
1
vote
0
answers
79
views
How to perform regression on data with uncertainty?
There are many resources for linear and polynomial regression, but I have not seen any material where the data comes with its own uncertainty as it appears in the real world. I have n data points, {...
- 41
4
votes
2
answers
3k
views
Uncertainty Quantification in Time Series Analysis
The stock market value of the data point connected by the red line is predicted by linear regression using market values as well as Twitter sentiment data and more in a certain period of time (red ...
- 2,051
3
votes
0
answers
97
views
Quantifying uncertainty of predictions for new data in the regression tree
I used Regression Learner to train my data. I held out 25% of the input for validation and ran different models for training. Based on the results using RMSE and R-squared, I decided to go for the ...
- 31
3
votes
0
answers
89
views
How to incorporate uncertainty and noise information in training and prediction of neural networks?
I am trying to use RNNs to perform state estimation on noisy sensor data. The readings are from a GPS dataset and it provides $[longitude, latitude, n_{satellites}]$. The last column, which is the ...
- 31
1
vote
1
answer
55
views
Reporting model uncertinty
I hope I'm using the right terms here. I've generated a statistical model (PLS regression) based on LWIR (8-10.5 micrometer) spectrum from some lab samples. This model predicts the concentration of a ...
- 227
0
votes
0
answers
156
views
Optimal subset from training data used in Random Forest
I have a set of say 10,000 spatial locations with associated values of a soil property (e.g. soil clay). In addition, I have 100 spatial covariates (e.g. elevation) which cover entirely my study area. ...
5
votes
0
answers
278
views
Quantifying uncertainty when fitting a statistical model to partial effects/dependencies from a random forest (or other machine learning model)
Question: I estimate the partial dependence of $y$ on one predictor in a fitted random forest (RF). I want to now fit a parametric model to this partial dependence. How can I estimate my uncertainty ...
- 20.4k
1
vote
0
answers
218
views
How to quantify uncertainty in nonparametric regression models
I'm trying to get a handle on what the current state of things is when it comes to quantifying uncertainty in nonparametric regression models. It seems like the options are
Use a Bayesian model and ...
- 159
1
vote
0
answers
465
views
Nested cross-validation and quantifying uncertainty
Background: I'm working on a ML project to predict a continuous target and am comparing different models using nested cross-validation, where I don't have access to the test set for which my model ...
- 753
1
vote
0
answers
81
views
Conveying uncertainty in accuracy measurements for machine learning models
I've noticed that depending on how I sample training and test samples I can get a range of model accuracies, but the mean of those accuracies is reasonable. Also for methods like random forests and ...
- 111
2
votes
1
answer
35
views
Combining frequent low quality estimates with infrequent high quality estimates
What are methods that combines/regress on frequent low quality estimates (laden with bias+noise) with infrequent high quality estimates to yield estimates that have lower uncertainty?
- 698
3
votes
0
answers
187
views
Propagating uncertainties using random forest out-of-bag accuracy estimates
Let's say I train a random forest on some data and get an out-of-bag accuracy estimate of 90%. I then predict a quantity using this trained forest. What should be the uncertainty I give to that ...
- 3,132
-1
votes
1
answer
197
views
Independent and conditionally independent
I was wondering if two variables can be independent and conditionally independent. For example, A and D are independent. But are they also independent given the evidence C? I think they are, because ...
- 151