Calibration can refer to predicting an explanatory variable from an observation of the dependent variable; to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc.
0
votes
0answers
3 views
Uncertainty of a method calibrated against known standards
I'm trying to determine the measurement uncertainty of a quantity I am measuring with a method that I am calibrating against a known standard. For example (with arbitrary units) :
Standard #1 :
...
0
votes
1answer
13 views
How do I scale/standardize one set of data that is non-linear (due to temp variations in a device) to that of a data set that is linear?
I have data from two instruments over time. One of the instruments does not show a linear trend due to an anomaly but I want to standardize that data against that of the second instrument because I ...
1
vote
0answers
15 views
Regression vs Calibration [duplicate]
When reading on the wikipedia page of Calibration they said :
" A reverse process to regression, where instead of a future dependent variable being predicted from known explanatory variables, a ...
0
votes
1answer
56 views
Probability Calibration messes Reliability
I have about 1000 samples with 20 features and I'm using Random Forest to predict a binary class.
I'm trying to apply the probability calibration process as described on scikit using ...
3
votes
0answers
32 views
Signatures of underfitting and overfitting in logistic regression calibration curves
My confusion stems from reading the following paper
http://www.bmj.com/content/351/bmj.h3868
It states in its abstract (and they later show an empirical study that conforms to the claim) - "...
1
vote
0answers
14 views
How can I perform a k-fold cross-validation for a calibrated (physical) model?
I have already read several forum contributions, but the general implementation of a cross-validation is still unclear to me (especially for the specific application).
The physical model was ...
0
votes
0answers
13 views
Poor and Random Validation results in stepwise regression
I'm doing predictor selection for downscaling from atmospheric predictors using step wise multiple regression during time period 1951-2005. I have split the whole data in two non-overlapping periods- ...
0
votes
0answers
8 views
maximum likelihood estimation for Generalized normal error distribution
In short, I want to understand why using Laplacian distribution to model errors performs better than using Gaussian distribution to model errors. In detail, I am using Generalized normal distribution ...
0
votes
0answers
7 views
Coefficient of variation: on raw vs calculated values
I have measured a quantity of say 60 replicate samples which correspond to a certain concentration of a chemical in the sample. These have been prepared using a certain protocol which I want to assess ...
1
vote
0answers
18 views
Only ROC reported for predictive model
I have just finished reading a medical paper that claims to have developed a predictive model but reported ROC alone without a calibration curve.
ROC seems to be useful for cases of binary ...
0
votes
0answers
7 views
How much independent data to hold out for calibration using `CalibrateClassifierCV`
One option for CalibratedClassifierCV (I use Platt scaling) is to manually specify which data will be used for fitting model parameters and which (independent) data ...
1
vote
0answers
22 views
Can a calibrated prediction model that is able to discriminate have a poor Brier score?
I did a logistic regression with selected covariates on a dataset with about 10000 records and event rate of 10%. The cross validated c-index was 63% which admittedly is not very high. Looking at a ...
1
vote
1answer
50 views
Help interpreting calibration curve
How is the "shortness" of the curve interpreted? I have seen studies that just crop this plot and declare "good calibration," but is it? The classifier never predicts above 0.6 in either the training ...
0
votes
0answers
27 views
Maximum probability returned by classifier much less than 1: truncated calibration curve
There's a sklearn calibration curve example which shows curves for different classifiers. I changed it to reproduce an issue I am having on a true dataset by ...
0
votes
0answers
11 views
Error in calibration lrm model using rms
I'm having difficulties to calibrate a logistic regression model using the rms package. Below is my R-script:
data = d
set.seed(1)
model <- lrm(y~x1+x2+x3, x=TRUE, y=TRUE, data=d)
validate(model, B=...
