All Questions
Tagged with uncertainty machine-learning
18 questions with no upvoted or accepted answers
5
votes
0
answers
278
views
Quantifying uncertainty when fitting a statistical model to partial effects/dependencies from a random forest (or other machine learning model)
Question: I estimate the partial dependence of $y$ on one predictor in a fitted random forest (RF). I want to now fit a parametric model to this partial dependence. How can I estimate my uncertainty ...
- 20.4k
3
votes
0
answers
97
views
Quantifying uncertainty of predictions for new data in the regression tree
I used Regression Learner to train my data. I held out 25% of the input for validation and ran different models for training. Based on the results using RMSE and R-squared, I decided to go for the ...
- 31
3
votes
0
answers
89
views
How to incorporate uncertainty and noise information in training and prediction of neural networks?
I am trying to use RNNs to perform state estimation on noisy sensor data. The readings are from a GPS dataset and it provides $[longitude, latitude, n_{satellites}]$. The last column, which is the ...
- 31
3
votes
0
answers
187
views
Propagating uncertainties using random forest out-of-bag accuracy estimates
Let's say I train a random forest on some data and get an out-of-bag accuracy estimate of 90%. I then predict a quantity using this trained forest. What should be the uncertainty I give to that ...
- 3,132
1
vote
0
answers
228
views
Probabilistic machine learning models: parameter uncertainty
Consider models such as DeepAR, ngboost and other frameworks to the general problem of predicting the parameters of some parametric distribution with some black-box function, call it f(X). The ...
- 831
1
vote
0
answers
71
views
Distributions as Features in Machine Learning
The Problem
Let's assume I have a problem that seems perfect for supervised learning. However, some of the measurements I would like to use as features are not point estimates but are instead ...
- 153
1
vote
0
answers
21
views
Measuring uncertainty of model prediction by repeat measurements
Say I’ve trained some single value regression ML model (a neural network or something). I have trained this ML model with simulation data. I see that this neural network is good at predicting data in ...
- 111
1
vote
0
answers
154
views
How to calculate uncertainty from test set size
I'm training a machine learning model and trying to determine how large my test set should be. I'm not using any k-fold cross-validation, just the test set. I believe the only benefit of increasing ...
- 438
1
vote
0
answers
88
views
Propagating uncertainty through nested random forest models
Does anyone know if there are methods for propagating the prediction intervals (i.e. uncertainty) of nested surrogate models, specifically random forests? When I say nested, I mean that a second model ...
- 11
1
vote
0
answers
79
views
How to perform regression on data with uncertainty?
There are many resources for linear and polynomial regression, but I have not seen any material where the data comes with its own uncertainty as it appears in the real world. I have n data points, {...
- 41
1
vote
0
answers
218
views
How to quantify uncertainty in nonparametric regression models
I'm trying to get a handle on what the current state of things is when it comes to quantifying uncertainty in nonparametric regression models. It seems like the options are
Use a Bayesian model and ...
- 159
1
vote
0
answers
465
views
Nested cross-validation and quantifying uncertainty
Background: I'm working on a ML project to predict a continuous target and am comparing different models using nested cross-validation, where I don't have access to the test set for which my model ...
- 753
1
vote
0
answers
81
views
Conveying uncertainty in accuracy measurements for machine learning models
I've noticed that depending on how I sample training and test samples I can get a range of model accuracies, but the mean of those accuracies is reasonable. Also for methods like random forests and ...
- 111
0
votes
0
answers
15
views
How to determin the theoretical prediction limit for a complex process?
how can we find out what is the theoretical prediction limit for a complex process?
For example, for a coin toss (on average) the prediction limit is 50%, that is we cannot predict better than this ...
- 73
0
votes
0
answers
59
views
What can you do with quantified uncertainty in latent variable time-series models?
Uncertainty quantification in latent variable models is a topic I am interested in, but I am struggling to grasp the difference between what you can do with quantified parameter uncertainty and ...
- 1
0
votes
1
answer
270
views
Evaluating the model stability using bootstrapping
I need help with the following.
Using our alternate data for external clients, we have built a model for identifying fraudulent customers (classification).
we used the auto-ml package to arrive at the ...
- 1
0
votes
0
answers
33
views
Uncertainty estimation in the input space
my input is an array between 0 and 1000 and the output is the corresponding system velocity. The input value is randomly generated (for instance by using the function in Python ...
- 101
0
votes
0
answers
156
views
Optimal subset from training data used in Random Forest
I have a set of say 10,000 spatial locations with associated values of a soil property (e.g. soil clay). In addition, I have 100 spatial covariates (e.g. elevation) which cover entirely my study area. ...
