Questions tagged [validation]

The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.

312 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
9
votes
0answers
1k views

How to evaluate performance of (variational) autoencoders?

Let's assume that you have trained your (variational) autoencoder on MNIST digits. After some time, you check the result and decide that the reconstruction is pretty good. But this is highly ...
9
votes
0answers
491 views

Can Frank Harrell's method be used to obtain optimism-corrected regression coefficients?

I used a regularized (LASSO) Cox regression to estimate relapse times of patients and used Frank Harrell's bootstrapping method to obtain an optimism-corrected performance estimate of my model. ...
5
votes
0answers
101 views

Official name of a common type of Bayesian simulation study

There is a kind of simulation study that is commonly used to validate an implementation of a Bayesian model: For independent replication $i = 1, ..., n$: Draw a set of "true" parameters ...
5
votes
0answers
6k views

How to validate a Poisson GLMM model?

I’m using the glmer function from the lme4 package in R to model species richness adjacent ...
5
votes
0answers
3k views

Hold out sample vs. cross validation for time series, and how to perform in R

I think out-of-sample validation testing for accuracy is essential in initially judging what time-series forecasts to use. In any case, I've been doing some reading on the two most common methods, ...
4
votes
1answer
3k views

Creating dummy variables before or after splitting to train/test datasets

I have a data set with a few columns of categorical type. As part of modelling process, I need to convert them into dummy variables. My confusions is whether to do dummy creation before or after ...
4
votes
0answers
2k views

Rolling window time series training and validation in Keras

I have a conceptual question regarding the use of the rolling window approach for training and validating a recurrent neural network (LSTM or GRU) on time series data. I have daily time series data ...
4
votes
0answers
129 views

Why are the predictions of my models getting worse?

I am creating a series of monthly models to predict the yearly production of grain for a fixed US state. For a given month, the model is built using the data from the past 10 years and it allows me to ...
4
votes
0answers
188 views

How do you evaluate machine learning model already deployed in production?

so to be more clear lets consider the problem of loan default prediction. Let's say I have trained and tested off-line multiple classifiers and ensembled them. Then I gave this model to production. ...
4
votes
0answers
810 views

How to Validate a Monte Carlo Simulation

I have historical data of a production process, and I've being asked to build a simulation model to predict its performance in the future. Using the historical data, I've being able to obtain the ...
4
votes
1answer
1k views

Splitting between train/test for customer churn survival models

I am a bit confused on how data can be split between train/test and "live" data for predicting churn using survival models such as the one in RandomForestSRC package. Goal of the model is to predict ...
4
votes
1answer
447 views

Computing prediction intervals with cross-validation?

I'm using a k-fold (10-fold) cross-validation while building a model. I'm only using it to get an estimate of the out-of-sample error, not to pick a model from candidates. For example, if I have 30 ...
4
votes
1answer
212 views

Sampling a test set from global spatial data

The basis of testing the accuracy of any machine learning algorithm is to test the trained algorithm on data that it has never seen before. The usual approach to sample the test set is to just ...
3
votes
1answer
37 views

Why do we choose the hyperparameters that gives the lowest validation error? Do we assume that it also gives the lowest generalization error?

The usual way of selecting hyperparameters is to tune it on the validation set and select the hyperparameters that gives the lowest validation error (Lets assume the validation sample is large so we ...
3
votes
0answers
69 views

How do you evaluate the prediction accuracy of linear mixed models?

How does one evaluate prediction accuracy with uncertainty for linear mixed models? Let's say I do bootstrapping and do train/test each time, and want to generate confidence intervals for some ...
3
votes
0answers
55 views

A goodness of fit test for two discrete distributions with unequal variance?

I have a computer simulation in which a virtual agent end up in different areas of a layout based on several factors. There are 18 conditions, so the data (you can find the csv file for a toy dataset ...
3
votes
0answers
37 views

When and how to re-evaluate deployed models

There are a lot questions about how to train a model (family) or how to tune hyperparmeter. But there are surprisingly few question about how to monitor or evaluate a model already in production (...
3
votes
0answers
654 views

How to build valid GAMLSS models?

Sorry for the following basic questions but it is important for me to get a feedback for my approach. I would like to create reference values for children in form of percentile curves (also called Z-...
3
votes
0answers
466 views

Understanding KS Statistic as Model Selection Tool

As a hobbyist learning about predictive modeling and machine learning, I am having some difficulty finding clarity regarding the KS statistic as a method for model selection. My mentor has been ...
3
votes
0answers
364 views

Training and Test Set Splits for Survival Analysis

Are there any special considerations for creating a training and testing split when working with survival analysis? My specific problem is to use 5 years of customer data to predict retention for ...
3
votes
0answers
182 views

Why Does My Model Accuracy Go Down With More Data?

I am using R and the caret package. My code is very straight forward. ...
3
votes
0answers
76 views

How to check whether there are any clusters in data?

In short: I am using k-means clustering with correlation distance. How to check, how many clusters should be used, if any? There are many indices and answers on how to establish a number of clusters ...
3
votes
0answers
804 views

Accounting for autocorrelation in predictive models?

Is it necessary to account for autocorrelation of data in a predictive model? I ask because it seems that accounting for autocorrelation (temporal or spatial) by means of ACF, PCF, etc. is typically ...
3
votes
0answers
152 views

External Validation for SVM

As important as I have found external model validation to be, there is certainly a lack of material out there. The closest thing I have found is a paper that is focused on external validation for a ...
3
votes
0answers
193 views

How to account for patterns in GAMM residuals due to the addition of an offset?

I have been wrestling with some model problems for a few weeks now and would like some help. I am trying to determine whether fish catch rates have changed following an event using GAMM. Because ...
3
votes
0answers
212 views

How to deal with floor effect

I am in the process of validating a five-items scale for measuring dependence on substance 'X'. I have collected data from 98 people who used the substance under consideration at least once weekly for ...
3
votes
0answers
507 views

How to compare and validate imputation models?

I've seen a lot of interesting questions here about multiple imputation and also great answers that helped me a lot to impute my data. I've used Predictive Mean Matching, EMB and I would like to use ...
3
votes
0answers
304 views

Validating statistical tests for value at risk and expected shortfall

I am trying to figure out if value-at-risk (VaR, a quantile) type tests could capture if expected shortfall (expectations above a quantile) point forecast generated from a type of model could be ...
3
votes
0answers
187 views

Validation of mixed-effect models

I want to use linear mixed effect model for a set of data. After using lme4 package and lmer() function and fitting model, I want to validate my model for other ...
3
votes
0answers
180 views

What is the criterion of model validation?

Suppose I have a model that boasts given 1 year's daily data to calibrate the parameters, it could predict the behavior of future 1 month. What should be the criterion for back-testing or, model ...
3
votes
0answers
313 views

Can holdout validation be systematically biased?

I recently did some experimenting comparing some common method of internal validation. In my field, the use of a single 1:1 holdout validation is extremely common, even with very small datasets, and I ...
3
votes
0answers
117 views

What tasks should I do when performing a model validation?

I have a mixed model built using Data A below, and now I want to validate the model using Data B. What should I do to actually "validate" the model? ...
3
votes
0answers
5k views

Validation error less than training error -- implications?

I am running a neural net to predict used car prices, sample size is 800. Using both 10-fold cross validation (10 times) and 1/3 holdback (10 times), the $R^2$ for training is about 0.60 and for ...
3
votes
1answer
2k views

Validation loss fluctuating while training the neural network in tensorflow

While training my convolutional neural network to predict emotions, I displayed at the same time the training and the validation data loss. The training loss appear to decrease over time, while on the ...
3
votes
3answers
10k views

Validation accuracy reach to 1.000 in very first epoch

I am using below small 3D CNN to predict whether 32*32*32 image cube in a CT scan is malignant or not. ...
3
votes
1answer
288 views

The effect of oversampling on the positive predictive value

I need to calculate the positive predictive value for a validation set for a rare event. The problem is that the validation set was oversampled for the rare event. The event occurs in 5 percent of the ...
2
votes
2answers
290 views

Why validation accuracy is increasing very slowly?

My convolutional network seems to work well in learning the features. However, the accuracy of the validation set is increasing very slowly with respect to the learning rate as also illustrated in the ...
2
votes
1answer
202 views

Shouldn't ROUGE-1 precision be equal to BLEU with w=(1, 0, 0, 0) when brevity penalty is 1?

I am trying to evaluate a NLP model using BLEU and ROUGE. However, I am a bit confused about the difference between those scores. While I am aware that ROUGE is aimed at recall whilst BLEU measures ...
2
votes
0answers
45 views

Can I use a (paired) t-test to compare the accuracies of classification models on a hold out test set?

When comparing the Brier scores or logloss/cross-entropy for multiple models on a hold-out test set, how can I be sure the 'best' model doesn't just have the lowest loss due to chance? When can I be ...
2
votes
2answers
36 views

What is statistical meaning of testing of machine learning model on new data?

Any machine learning model, normally, is applied on new data for testing. If the accuracy of the model is below a certain threshold, the model is rejected. But why? This is just a random sample, the ...
2
votes
0answers
20 views

What happens if validation/real data has values outside training set limits for continuous data?

Describing Example: If feature X1 in training data has values inside [0,1] However, X1 in real/validation data has values [-1, 2[ What happens then? Previously discussed On previous discussions with ...
2
votes
1answer
103 views

How to validate a regression model in extrapolation territory?

I'm dealing with a regression problem and have two datasets at my disposal. Dataset A is properly labeled and I use it to fit and validate my model, B is unlabeled and I can only visually inspect ...
2
votes
0answers
88 views

Any methods for sanity checks on refreshed machine learning models?

I am wondering if there are any best practices for validating ML models that are trained on new data. Apart from the validation metrics used on test data, are there any other recommended approaches ...
2
votes
1answer
120 views

Validating uncertainty quantification

Regression performance is often evaluated by means of cross-validation. However, classical cross-validation only regards the mean of the identified parameters. How can one quantify the quality of the ...
2
votes
0answers
141 views

OOB error prediction in RF if case weights are used

I have a dataset for which grossing-up factors are given. I am using these factors as case weights for a random forest (R package ranger). Until now I was using the OOB prediction error for tuning, ...
2
votes
1answer
437 views

Validation accuracy/loss goes up and down linearly with every consecutive epoch

I'm training a CNN in keras with tensorflow backend with the following model architecture for a binary classification problem. I've divided approximately 41k images into training, validation and test ...
2
votes
0answers
695 views

Accuracy and ROC for Logistic and Decision Tree

So I run a logistic regression and decision tree model using same data. The accuracy shows that the decision tree outperforms logistic slightly. However, my ROC curve shows that logistic is much ...
2
votes
0answers
174 views

Optimism bootstrap with non-linear models

I have come across an example in my research with heavily overfit non-linear probabilistic classifiers, where the optimism bootstrap appears to underestimate the optimism, even when using a proper ...
2
votes
0answers
86 views

How to test whether a prediction interval truly captures 95%?

I want to analyze the 95% prediction intervals for a model. The true values should fall within the prediction intervals 95% of the time (on average) if the interval is well calibrated. If the ...
2
votes
0answers
10 views

Is it possible to "tune" a trained model in one population so it can be used for a different population (i.e., by swapping variables)?

Say, I have trained a model to classify patients into cardiovascular disease (CVD) and non-CVD. The model building process is as follows: There is a gold standard to compare the model with. The ...

1
2 3 4 5
7