Questions tagged [validation]
The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.
320
questions with no upvoted or accepted answers
2
votes
0
answers
332
views
Aggregating ROC AUC values of several Logistic Regression Models
I have a dataset that consists of six different segments. I have calculated a Logit Regression Model for each of those segments (binary response variable, 30.000 observations in total, 63 variables ...
- 1,109
2
votes
0
answers
200
views
Optimism bootstrap with non-linear models
I have come across an example in my research with heavily overfit non-linear probabilistic classifiers, where the optimism bootstrap appears to underestimate the optimism, even when using a proper ...
- 31
2
votes
0
answers
93
views
How to test whether a prediction interval truly captures 95%?
I want to analyze the 95% prediction intervals for a model. The true values should fall within the prediction intervals 95% of the time (on average) if the interval is well calibrated. If the ...
- 123
2
votes
0
answers
11
views
Is it possible to "tune" a trained model in one population so it can be used for a different population (i.e., by swapping variables)?
Say, I have trained a model to classify patients into cardiovascular disease (CVD) and non-CVD. The model building process is as follows:
There is a gold standard to compare the model with. The ...
- 1,167
2
votes
0
answers
18
views
What should I do when individual variables don't validate in a test set?
I built a model for purposes of prediction that validates out well overall on a test set. However, when looking at individual variables, there appears to be a couple (out of 100) that show no pattern. ...
- 81
2
votes
0
answers
124
views
Information leaks related to test or validation sets
Lately I've been reading about (indirect) information leaks, related to validation and tests sets in the context of hyperparameter tuning.
For example, Prakhar Agarwal, in his answer Does my ...
2
votes
0
answers
2k
views
Confusing aspect of Harrell's bootstrap-based optimism-correction Internal Validation procedure
This bootstrap-based internal validation procedure is implemented in the validate function in Harrell's rms package. It allows ...
- 46
2
votes
0
answers
1k
views
Early stopping criteria when training neural networks
What happens if the early stopping criteria suggest to stop training at a very early stage (i.e. after 10 epochs or so). Is it an indicator that more regularization should be applied to the model (I ...
- 987
2
votes
0
answers
29
views
Is there an official rule, or a generally accepted one, for how closely your validation data should match your training data?
I'm finding myself making a gut-feeling judgment more and more often on whether the validation of my model is "close enough" to the modeled results from my training data. I don't recall having ever ...
- 21
2
votes
0
answers
149
views
Discriminant validity between traits that co-vary?
I'm hoping to do a validation study on a measure of anxiety (called the AISCs) in a stroke population, using convergent and divergent validity.
A test demonstrates convergent validity if it ...
- 95
2
votes
0
answers
845
views
online vs offline model performance
I work in a group that deploys models that are trained offline and evaluated offline before being deployed. THis is for an ads ranking system.
We also record whether the user clicked on our ...
2
votes
0
answers
281
views
Optimism bias - alternative references
In Hastie & al's book Elements of Statistical Learning, there are two subsections covering insample prediction errors and optimism bias (section 7, p.228-230).
Hastie & al explain that ...
- 41
2
votes
0
answers
417
views
Evaluate image segmentation with the absence of ground truth
Motivation:
- Evaluate the computerised image segmentation against manual segmentation;
- Evaluate the difference between difference manual segmentation.
Background:
Given a raw medical image (...
- 41
2
votes
0
answers
99
views
What are good metrics for evaluating inverse covariance matrix?
For real datasets, where it's impossible to know the true inverse covariance, what are the methods of evaluating your inverse covariance estimator?
Possible answers:
If the number of features is ...
- 1,277
2
votes
1
answer
477
views
Score/validate a regression Neural Network
I wonder what is the best statistical method to validate/score/evaluate a regression Neural Network used to predict probabilities (an example would be using a Regression NN to predict the probability ...
- 121
2
votes
0
answers
97
views
Active learning system confusion matrix
A colleague of mine has been developing a machine learning system with an active learning component. I was having trouble reproducing the metrics he's been reporting, until I found out that he's ...
2
votes
0
answers
1k
views
What is the formula of Median Magnitude of Relative Error? (MdMRE)
I'm familiar with these terms
1 - MRE Mean Relative Error
2 - MMRE Mean Magnitude of Relative Error
I need to compute MdMRE which is Median Mean Relative Error. I searched on the net but didn't ...
2
votes
0
answers
562
views
Using likelihood to compare model accuracy on two different data sets
I have a model that produces conditional probabilities, $p(Y|X)$, where $Y$ is either 0 or 1, and $X$ is just some random variable.
I have two different data sets $Z_1, Z_2$ consisting of pairs of ...
- 39
2
votes
0
answers
183
views
Topic Modeling Dataset for Code Verification
I am trying to write up a Gibbs sampling Latent Dirichlet Allocation function for myself in R, and wanted to run it on a dataset where the true classification of ...
- 15.6k
2
votes
2
answers
509
views
What test shall I use to validate the use of a certain score to predict my outcome in a survival analysis?
I validate usage of a clinical cardiovascular score to predict the risk of dementia using data from a longitudinal study. Therefore, my outcome is binary (dementia yes or not) and the independent ...
- 263
2
votes
0
answers
155
views
What should I do when logistic regression model doesn't perform well on test sample?
For logistic model, I divided dataset into two parts training sample (70% of 360 data point (observation)) and test sample (rest 30% of 360) randomly. After that I built logistic model on training ...
- 429
2
votes
1
answer
597
views
Sample size needed to validate classification/prediction model
Dose any rule of thumb exist (or possible calculation) regarding sample size needed to validate an binary classification model.
We have developed this prediction model for a medical condition and ...
- 41
2
votes
0
answers
4k
views
Training/ Test Data with Time Series Model -- Forecast with Training Model, or with Model based on Full Data?
Okay, I have a couple books on time series forecasting, but perhaps I need to read a couple more.
Here's my question.
You want to be able to validate a forecasting model. So you split the data into "...
- 395
2
votes
0
answers
33
views
Looking for a Good Text on Statistical Analysis of Satisfaction Data
I am looking for a good textbook (or other resource) that covers the analysis of satisfaction data. Most of my data uses likert-type scales. Can anyone recommend something with examples?
- 41
2
votes
0
answers
86
views
Tuning a model for predictive performance in a narrow probability range
Assume we model the probability of disease incidence. When an individual's predicted probability of incident disease, or absolute risk, is greater than a certain threshold, we start preventive ...
- 3,684
2
votes
0
answers
369
views
Model validation in Bayesian statistics from a model with latent variables
I am working with some two-regime autoregressive models first introduced by Hamilton in 1989. The specific models is of no great concern to my question, but some variables within my autoregressive ...
- 1,734
2
votes
0
answers
127
views
Ways to compare ordered binary datasets?
This question regards the best way to compare ordered binary data. My situation is as follows:
I'm interested in evaluating how well a model conforms to human performance data on a set of validation ...
- 41
2
votes
0
answers
1k
views
Kendall's tau for holdouts low and not significant - Conjoint
I have done my Conjoint Analysis (fractional factorial design) but when it comes to validating the model, it shows a Kendall's tau for Holdouts of 0.33 and not significant.
But Pearson's R and Kendall'...
- 23
2
votes
0
answers
226
views
Validation of flow cytometer (Facs Canto II)
I was just wondering if 3 biological samples would be statiscally significant for both precision testing and calibration verification.
Precision: to check whether instrument is able to give ...
- 21
1
vote
0
answers
7
views
Image classification metrics
I have been working on an image classification task using CNNs and getting some puzzling results.
My training, validation and test loss keep going down with epochs and are comparable. So this might ...
- 11
1
vote
0
answers
40
views
How to statistically validate samplers? Is Chi-Square test relevant for synthetic data?
I am developing sampling procedures for different kinds of discrete objects (rankings, sets, etc.). In most cases I know the theoretical distribution of the objects I'm sampling. My idea is thus to ...
- 11
1
vote
0
answers
12
views
what does it mean to have slightly higher valid error rate, very high and similar valid error rate compared to train and test?
the dataset is split into 3 categories: train, test, valid. after training the error rates we see have these values.
case 1
1% error on the training set.
5% error on training dev set.
5% error on the ...
- 111
1
vote
0
answers
86
views
Unusual results from XGBoost learning curves
I'm working on training an XGBoost classification model on time series data. Currently, I have a lot of data and it is hard to fit it all in memory, so I am trying to better understand if more data ...
- 21
1
vote
0
answers
41
views
On the reliability of validation loss as a metric
I have the following plot of the training and validation loss from a deep neural network. The signature U-curve of the validation loss can be noticed. I want to use the validation loss as the metric ...
- 777
1
vote
0
answers
37
views
Do I need statistical test with nested-cv?
I'm working with a small data set and with 4 algorithms. The optimization process showed to be Very important as long as it improves a lot their performances. So, I'm using 10x5 nested-cv to estimate ...
1
vote
0
answers
54
views
Clarification for the test and validation set
I have been trying to figure out the difference between validation and test set used in the machine learning studies. In almost all places, I found that I have to split the dataset into three parts (...
- 33
1
vote
1
answer
27
views
Why the validation error does not decrease with a training of 30 epochs but decreases with a training of 60 epochs
When i train my model with 30 epochs, the training and validation error curves seems to stagnate:
However, when i train my model with 60 epochs, the training and validation error curves start to ...
- 11
1
vote
1
answer
17
views
What is the correct term to describe the difference between the predicted values and testing values in machine learning?
We generally use the term residuals to refer to the difference between the training data values and the model's predicted values, but I wonder if there is a different/better term to refer to the ...
- 347
1
vote
0
answers
14
views
Do I need to validate a subscale I made in SPSS from a previously validated scale?
I am using a commonly used, validated scale on positive and negative affect. If I want to look at the effect of the negative items in isolation, does the subscale I make of the negative items need to ...
- 11
1
vote
1
answer
54
views
What can be a robust model validation strategy for time series modelling?
I am working on time series data where I am using FB Prophet algorithm. I am willing to build a dashboard to produce validation results to my clients. What could be a possible model validation ...
- 11
1
vote
0
answers
27
views
Validation Metrics Peaking to Extreme Values
I am training a 3D Unet model with MAE loss to synthesis MRI scans from CT scans. The learning rate is 10e-3, and I am running it with a batch size of 6 through 600 epochs. The validation loss looks ...
1
vote
0
answers
344
views
Model validity for Ordinal logistic model
I am doing a study using OLR. The model tries to assess the satisfaction of ground level stakeholders (scale of 1(extremely dissatisfied) to 5(highly satisfied) in an urban area. The independent ...
- 11
1
vote
0
answers
81
views
How to validate a clustering model on new data?
I was wondering how one would properly validate the reproducibility of a trained clustering model on a completely new set of data.
Imagine you are clustering a patient population in hospital A. You ...
- 11
1
vote
1
answer
127
views
Fluctuations in both the accuracies and losses in training and validation of Deep learning MLP
I have a binary classification problem with Dataset N430 and predictors=146. Both Validation and training accuracies along with losses fluctuates. What would be the reason and suggest solution please?
- 11
1
vote
0
answers
109
views
consistent gap between training and validation metrics
I am training a neural network (Deep and cross network) for a multi-label classification task (~700 labels). I am seeing a uniform gap between training and validation results on various metrics. E.g. ...
- 311
1
vote
0
answers
65
views
Do multiple measurements on each patient require mixed effect methods if the number of measurements per patient is the same?
For the purposes of this question, let's acknowledge that accuracy is not the best way to measure a model's performance, and that a proper scoring rule is best. However, the decision to use accuracy ...
- 36.5k
1
vote
0
answers
84
views
Approximately Unbiased P-value vs Bootstrap Probability: which one should i choose?
Some references first:
How is approximately unbiased bootstrap better than a regular bootstrap with regards to hierarchical clustering?
Suzuki et al. 2004 https://www.researchgate.net/publication/...
- 133
1
vote
1
answer
76
views
Reliable methods to validate clustering of text phrases?
Question is in the title. I have clustered the word embeddings of text phrases, and now want to try and check whether the resulting clusters are coherent enough. I have tried methods that are ...
- 11
1
vote
0
answers
40
views
How do I perform a train-validation split on data with class imbalance such that the class imbalance ratio is preserved?
My data has class imbalance-- that is, some classes have significantly fewer training samples than the others. I want to perform a train-validation split in such as way that the class ratios are ...
- 23
1
vote
0
answers
101
views
Validation accuracy does not reflect the actual change in the testing accuracy
When training a CNN classifier for the Fashion-MNIST, I have noticed that there are multiple instances that validation error is not improving even if the tesing accuracy is improving every epoch or ...
- 777