Questions tagged [validation]

The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.

Filter by
Sorted by
Tagged with
516
votes
11answers
590k views

What is the difference between test set and validation set?

I found this confusing when I use the neural network toolbox in Matlab. It divided the raw data set into three parts: training set validation set test set I notice in many training or learning ...
53
votes
3answers
32k views

How to select a clustering method? How to validate a cluster solution (to warrant the method choice)?

One of the biggest issue with cluster analysis is that we may happen to have to derive different conclusion when base on different clustering methods used (including different linkage methods in ...
68
votes
12answers
91k views

Hold-out validation vs. cross-validation

To me, it seems that hold-out validation is useless. That is, splitting the original dataset into two-parts (training and testing) and using the testing score as a generalization measure, is somewhat ...
6
votes
1answer
2k views

Cross-validation techniques for time series data

What is an appropriate cross-validation technique for time series data? I have a daily 4 years time series data and fitting a SVM model by MATLAB R2015b: ...
30
votes
2answers
4k views

Should final (production ready) model be trained on complete data or just on training set?

Suppose I trained several models on training set, choose best one using cross validation set and measured performance on test set. So now I have one final best model. Should I retrain it on my all ...
17
votes
2answers
10k views

What is the procedure for "bootstrap validation" (a.k.a. "resampling cross-validation")?

"Bootstrap validation"/"resampling cross-validation" is new to me, but was discussed by the answer to this question. I gather it involves 2 types of data: the real data and simulated data, where a ...
6
votes
1answer
9k views

How to correctly use validation and test sets for Neural Network training?

I am in the machine learning business for a long time, but still, this fundamental fact gets me confused, since every paper, article and/or book describe different kind of usages for validation and ...
5
votes
3answers
12k views

Computing c-index for an external validation of a Cox PH model with R

First off, I'll state that I'm aware many questions get asked about the c-index. I've searched this site and others, and I haven't found an answer for my situation. I can successfully use ...
41
votes
4answers
40k views

How do you use the 'test' dataset after cross-validation?

In some lectures and tutorials I've seen, they suggest to split your data into three parts: training, validation and test. But it is not clear how the test dataset should be used, nor how this ...
40
votes
2answers
7k views

How to draw valid conclusions from "big data"?

"Big data" is everywhere in the media. Everybody says that "big data" is the big thing for 2012, e.g. KDNuggets poll on hot topics for 2012. However, I have deep concerns here. With big data, ...
16
votes
1answer
3k views

Name of mean absolute error analogue to Brier score?

Yesterday's question Determine accuracy of model which estimates probability of event got me curious about probability scoring. The Brier score $$\frac{1}{N}\sum\limits _{i=1}^{N}(\text{prediction}_i -...
3
votes
1answer
2k views

Interplay between early stopping and cross validation

I am a little bit confused by early stopping and in particular by how it can be inserted inside a CV framework. As far as I understand, I can fix the optimal number of epochs (for NN, or number of ...
3
votes
1answer
926 views

Cross-validation for Comparing Clustering Techniques

I'm working on comparing multiple clustering algorithms to each other using the adjusted Rand index for a given dataset. We have a gold standard that we'd like to compare the obtained clustering ...
5
votes
0answers
87 views

Official name of a common type of Bayesian simulation study

There is a kind of simulation study that is commonly used to validate an implementation of a Bayesian model: For independent replication $i = 1, ..., n$: Draw a set of "true" parameters ...
2
votes
1answer
1k views

Validation of logistic regression - goodness of fit (pearson)

I have developed a scoring system using logistic regression. The score ranges between 0 and 6 (using integers) and predicts death. It does not use a conventional regression formula and thus I am not ...
32
votes
3answers
21k views

Do we need a test set when using k-fold cross-validation?

I've been reading about k-fold validation, and I want to make sure I understand how it works. I know that for the holdout method, the data is split into three sets, and the test set is only used at ...
23
votes
4answers
7k views

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
11
votes
3answers
15k views

What is a consistency check?

I was asked such a question as "Did you do any consistency check in your daily work?" during a phone interview for a Biostatistician position. I don't know what to answer. Any information is ...
13
votes
2answers
931 views

Is the Error rate a Convex function of the Regularization parameter lambda?

In choosing the regularization parameter lambda in Ridge or Lasso the recommended method is to try different values of lambda, measure the error in the Validation Set and finally chose that value of ...
9
votes
4answers
2k views

Calculating ratio of sample data used for model fitting/training and validation

Provided a sample size "N" that I plan on using to forecast data. What are some of the ways to subdivide the data so that I use some of it to establish a model, and the remainder data to validate the ...
6
votes
2answers
3k views

Validity of pseudo-panel data constructed from repeated cross sectional data as a panel data

I am looking at the repeated cross-sectional data from federal reserves, which has both panel data and repeated cross sectional data at different time-points,e.g. 2007-2009 is a panel while 2010 is a ...
10
votes
1answer
7k views

Best practices for measuring and avoiding overfitting?

I am developing automated trading systems for the stock market. The big challenge has been overfitting. Can your recommend some resources describing methods for measuring and avoiding overfitting? I ...
6
votes
2answers
1k views

How to do external validation of regression models

Very basic question here, so bear with me... I have a data set with 241 patients with 16 variables plus diagnosis (malignant vs benign). There are 3 previously published logistic regression formulas ...
4
votes
1answer
9k views

How to validate Cox Proportional Hazards model?

I'm using a Cox proportional Hazards regression (R survival package) to model Credit card activation propension, ie, which people are more likely to make their first buy? To give more context: ...
5
votes
1answer
8k views

Model validation after fitting a negative binomial GLM in R

Ok, I have searched and searched and just have no clue where to start. First, what I would like to do is produce a QQ-plot (or even a readable residual plot) to look at the fit of my model. I guess ...
4
votes
3answers
17k views

How to validate a Multinomial Logit and Probit Model fit?

I would like to know how do you determine the performance of your models. That is, if you fit a multinomial logit or probit model for un-ordered discrete choice. What do you use to evaluate whether ...
4
votes
1answer
455 views

Does retraining a model on all available data necessarily yield a better model?

A (simplified) typical workflow in machine learning might be: Train $m$ models on a training set. Validate the $m$ models on a validation set to yield the best model with parameters $\theta$. Retrain ...
2
votes
2answers
1k views

Choosing best training-validation split?

In machine learning context, suppose I have 100 observations which will be split into training and validation set (say #1 ~ #100) and totally separate 100 observations for test set (say #101 ~ #200). ...
3
votes
2answers
9k views

Logistic Regression Model Validation

I am validating a logistic regression model. This is the first time i am validating a model. I am using split sampling method. I have split data randomly into two parts - 70% development and 30% ...
2
votes
2answers
164 views

outer folds errors in nested cross-validation

I have a time series data that I wish to be able to obtain the general performance of it. For that, I use nested cross-validation with time series flavor as described in this amazing blog. As you ...
1
vote
1answer
169 views

fitting after training and validation

There are a lot written in StackExchange about train-validation-test split of data set. I am confuse with the following. Assume, I trained model using train set. Then I choose model using validation ...
23
votes
2answers
16k views

Scikit correct way to calibrate classifiers with CalibratedClassifierCV

Scikit has CalibratedClassifierCV, which allows us to calibrate our models on a particular X, y pair. It also states clearly that ...
24
votes
2answers
2k views

Bayesian thinking about overfitting

I've devoted much time to development of methods and software for validating predictive models in the traditional frequentist statistical domain. In putting more Bayesian ideas into practice and ...
40
votes
3answers
4k views

Why is it that my colleagues and I learned opposite definitions for test and validation sets?

In my master's program I learned that when building a ML model you: train the model on the training set compare the performance of this against the validation set tweak the settings and repeat steps ...
17
votes
3answers
16k views

Splitting Time Series Data into Train/Test/Validation Sets

What's the best way to split time series data into train/test/validation sets, where the validation set would be used for hyperparameter tuning? We have 3 years' worth of daily sales data, and our ...
12
votes
3answers
3k views

Why isn't the holdout method (splitting data into training and testing) used in classical statistics?

In my classroom exposure to data mining, the holdout method was introduced as a way of assessing model performance. However, when I took my first class on linear models, this was not introduced as a ...
10
votes
2answers
23k views

How to make representative sample set from a large overall dataset?

What are the statistical techniques to create a sample set, which is representative of the entire population (with a known confidence level)? Also, How to validate, if the sample fits the overall ...
6
votes
1answer
24k views

Difference between training, test and holdout set data mining model building

What is the difference between training, test, and holdout sets? I know these concepts, just want to ensure that I have understood correctly. Training set is something that we have as of now. We ...
13
votes
1answer
5k views

What is the intuition behind the variation of information (VI) metric for cluster validation?

For non-statisticians like me, it is very difficult to capture the idea of VI metric (variation of information) even after reading the relevant paper by Marina ...
4
votes
1answer
1k views

Splitting between train/test for customer churn survival models

I am a bit confused on how data can be split between train/test and "live" data for predicting churn using survival models such as the one in RandomForestSRC package. Goal of the model is to predict ...
1
vote
2answers
1k views

Which balance strategy to learn from my very imbalanced dataset?

I'm using a deep learning approach on a dataset made of ~20 millions of elements, where each element has a TRUE or FALSE label. This dataset unfortunately is veeery imbalanced: I've 98% of falses and ...
17
votes
5answers
8k views

Can increasing the amount of training data make overfitting worse?

Suppose I train a neural network on dataset A and evaluate on dataset B (that has a different feature distribution than dataset A). If I increase the amount of data in dataset A by a factor of 10, is ...
10
votes
2answers
3k views

What is the difference between sensitivity analysis and model validation?

I read both wikipedia pages of sensitivity analysis and model validation (here, only linear regression validation) but I don't manage to find a way to separate these two terms. I have the impression ...
9
votes
3answers
7k views

Is overfitted model with higher AUC on test sample better than not overfitted one

i am participating in a challange in which I have created a model that performs 70% AUC on train set and 70% AUC on hold-out test set. The other participant has created a model that performs 96% AUC ...
6
votes
2answers
2k views

Logistic regression performs better on validation data

Recently I've been building a model using logistic regression. To my suprisise LIFT chart looks better on the validation data than on the training data, the same is with ROC. All variables in the ...
5
votes
2answers
44k views

How to determine the accuracy of logistic regression in R?

I'm curious about how to understand the accuracy of my model which I computed with glm( family = binomial(logit) ). In some articles it is mentioned that we should ...
5
votes
0answers
3k views

Hold out sample vs. cross validation for time series, and how to perform in R

I think out-of-sample validation testing for accuracy is essential in initially judging what time-series forecasts to use. In any case, I've been doing some reading on the two most common methods, ...
4
votes
1answer
3k views

Predicting customer churn - train & test sets

I'm struggling with a problem where I'm trying to predict customer churn. I have monthly snapshot data going back several years, and tags for whether a customer left during a given month. My main ...
3
votes
1answer
3k views

What are core statistical model validation techniques?

I am self-taught machine-learning Data Science enthusiast. Over the course of self-learning, I have come across various validation techniques such as LOOCV, K-fold cross-validation, the bootstrap ...
3
votes
1answer
1k views

Determining values of correction factor based on x bins in observed vs. actual data

I am trying to automate a problem I usually solve by hand. I have a sensor that collects data from the field. Every 6 months or so, I have to do a calibration on that sensor by collecting ...