Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

137
votes
9answers
129k views

What is the difference between test set and validation set?

I found this confusing when I use the neural network toolbox in Matlab. It divided the raw data set into three parts: training set validation set test set I notice in many training or learning ...
32
votes
2answers
4k views

How to draw valid conclusions from “big data”?

"Big data" is everywhere in the media. Everybody says that "big data" is the big thing for 2012, e.g. KDNuggets poll on hot topics for 2012. However, I have deep concerns here. With big data, ...
22
votes
4answers
769 views

As a reviewer, can I justify requesting data and code be made available even if the journal does not?

As science must be reproducible, by definition, there is increasing recognition that data and code are an essential component of the reproduciblity, as discussed by the Yale Roundtable for data and ...
21
votes
5answers
11k views

Hold-out validation vs. cross-validation

To me, it seems that hold-out validation is useless. That is, splitting the original dataset into two-parts (training and testing) and using the testing score as a generalization measure, is somewhat ...
11
votes
3answers
1k views

Why isn't the holdout method (splitting data into training and testing) used in classical statistics?

In my classroom exposure to data mining, the holdout method was introduced as a way of assessing model performance. However, when I took my first class on linear models, this was not introduced as a ...
11
votes
1answer
131 views

When building a regression model using separate modeling/validation sets, is it appropriate to “recirculate” the validation data?

Suppose I've got an 80/20 split between modeling/validation observations. I've fit a model to the modeling data set, and I'm comfortable with the error that I'm seeing on the validation data set. ...
10
votes
2answers
4k views

What is the procedure for “bootstrap validation” (a.k.a. “resampling cross-validation”)?

"Bootstrap validation"/"resampling cross-validation" is new to me, but was discussed by the answer to this question. I gather it involves 2 types of data: the real data and simulated data, where a ...
9
votes
3answers
5k views

What is a consistency check?

I was asked such a question as "Did you do any consistency check in your daily work?" during a phone interview for a Biostatistician position. I don't know what to answer. Any information is ...
9
votes
4answers
1k views

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
8
votes
2answers
361 views

Should final (production ready) model be trained on complete data or just on training set?

Suppose I trained several models on training set, choose best one using cross validation set and measured performance on test set. So now I have one final best model. Should I retrain it on my all ...
8
votes
1answer
2k views

What is the intuition behind the variation of information (VI) metric for cluster validation?

For non-statisticians like me, it is very difficult to capture the idea of VI metric (variation of information) even after reading the relevant paper by Marina ...
7
votes
3answers
413 views

How to select or validate the selection of a clustering method?

One of the biggest issue with cluster analysis is that we may happen to have to derive different conclusion when base on different clustering methods used (including different linkage methods in ...
7
votes
2answers
216 views

How to make sure that a machine learning algorithm's implementation is correct?

Say there is a machine learning algorithm (e.g. classification) that is well known and implemented by the original creators of the algorithm. Yet all you have is the ability to use the algorithm but ...
7
votes
1answer
1k views

Within-group sum of squares of cluster

I have a multivariate dataset for which I have only a table including the cross-wise Euclidean distances between all points and a list giving the assignment of each point to one of several clusters. ...
6
votes
1answer
543 views

Name of mean absolute error analogue to Brier score?

Yesterday's question Determine accuracy of model which estimates probability of event got me curious about probability scoring. The Brier score $$\frac{1}{N}\sum\limits _{i=1}^{N}(prediction_i - ...
6
votes
1answer
2k views

Best practices for measuring and avoiding overfitting?

I am developing automated trading systems for the stock market. The big challenge has been overfitting. Can your recommend some resources describing methods for measuring and avoiding overfitting? I ...
6
votes
1answer
94 views

How to account for case weights when generating folds for K-fold CV?

I am currently working on a binary classification problem where each point in the dataset is paired with a case weight. Specifically, each record $(w_i, x_i, y_i)$ represents a distinct type of person,...
6
votes
2answers
675 views

Internal validation via bootstrap: What ROC curve to present?

I am using the bootstrap approach for internal validation of a multivariate model built with either standard logistic regression OR elastic net. The procedure I use is as follows: 1) build model ...
5
votes
3answers
297 views

Is overfitted model with higher AUC on test sample better than not overfitted one

i am participating in a challange in which I have created a model that performs 70% AUC on train set and 70% AUC on hold-out test set. The other participant has created a model that performs 96% AUC ...
5
votes
1answer
97 views

What can be inferred from this residual plot?

This is with reference to http://analyticspro.org/2016/03/05/r-tutorial-residual-analysis-for-regression/. For Residual plot (b), where the residuals are increasing linearly (with respect to the ...
5
votes
2answers
385 views

How to validate if a sample is independent and identically distributed

How can I check if the data is drawn i.i.d. from an unknown multivariate distribution?
5
votes
1answer
124 views

What do you do with your testing data?

Suppose you perform a parameter selection study in an environment with not-so-many data points. You divide the available data into training and testing set and you plan to validate the future model on ...
5
votes
2answers
10k views

How to make representative sample set from a large overall dataset?

What are the statistical techniques to create a sample set, which is representative of the entire population (with a known confidence level)? Also, How to validate, if the sample fits the overall ...
5
votes
2answers
734 views

How to do external validation of logistic regression models and perform model benchmarking

Quality assessment in trauma has for > 25 years been done with the US derived logistic regression model, the TRISS model. DV: survival/death and IVs: physiologic derangement (continuous), anatomic ...
5
votes
2answers
2k views

Validate cluster analysis in R

I am trying to validate hierarchical cluster analysis result following a paper by Guy Brock, et al. clValid: An R Package for Cluster Validation (pdf). Do I have to use all these methods? What are the ...
5
votes
1answer
136 views

Validating a logistic regression for a specific $x$

I have a logistic regression model for 0/1 binary response data that is built from samples $(x_1,Y_1),\dots,(x_m,Y_m)$, where $x_1,\dots,x_m$ are, fixed, nonrandom, real values and $Y_1,\dots,Y_m$ are ...
5
votes
0answers
152 views

Are world cup predictions testable?

As of today, dozens of soccer world cup predictions exist, some more complex, some more elegant, and most of them predict every nation's "chance" of winning a particular match/ the cup. As I am ...
5
votes
1answer
499 views

Prediction evaluation metric for panel/longitudinal data

I would like to evaluate several different models that provide predictions of behavior at a monthly level. The data is balanced, and $n=$100,000 and $T=$12. The outcome is attending a concert in a ...
4
votes
2answers
2k views

How exactly to partition training-set for k-fold cross validation on multi-class dataset?

Cross validation is one of the most important tools because it gives us an honest assessment of the true accuracy of our system. In other words, the cross-validation process provides a much more ...
4
votes
4answers
131 views

Should I get 100% classification accuracy on training data?

I've been getting inconsistent results with a binary classification problem I'm trying to solve using a linear classifier and a custom feature extraction pipeline, and decided to do a quick check of ...
4
votes
2answers
722 views

In logistic regression, does the lack of significance of the parameter estimates in a test sample indicate overfitting?

I am trying to build a logistic regression model where I have a dependent variable $y$ and independent variables $x_1$, $x_2$... $x_n$. $y$ can take only two values - 0 or 1. My original modelling ...
4
votes
2answers
418 views

How to do external validation of regression models

Very basic question here, so bear with me... I have a data set with 241 patients with 16 variables plus diagnosis (malignant vs benign). There are 3 previously published logistic regression formulas ...
4
votes
2answers
549 views

What is a good way to test a simple Recurrent Neural Network

I have coded up a simple real-value regression RNN in theano. What kind of dataset should I test it on? How should I go about testing it? My structure is: Univariate (for now) timeseries, $x_{...
4
votes
1answer
4k views

Model validation after fitting a negative binomial GLM in R

Ok, I have searched and searched and just have no clue where to start. First, what I would like to do is produce a QQ-plot (or even a readable residual plot) to look at the fit of my model. I guess ...
4
votes
2answers
628 views

Logistic regression performs better on validation data

Recently I've been building a model using logistic regression. To my suprisise LIFT chart looks better on the validation data than on the training data, the same is with ROC. All variables in the ...
4
votes
2answers
2k views

Using Adaboost for feature selection?

Is it okay to use Adaboost to do feature selection (selecting a subset of dimensions $S$ from a high-dimensional feature vector $V$)? I divided the samples into four non-overlapping sets: $A$ (...
4
votes
1answer
834 views

Best method to validate a multiply imputed Cox model with R?

This question is with regards to using a test data set to validate an imputed Cox model using R. With a non-imputed data set I would use val.surv() from ...
4
votes
2answers
1k views

Variance explained of a mixed effects model in a new data set

I'm using the following function to calculate Edwards R^2 (formula 19 in Edwards et al. 2008) of a mixed effects model (I hope the implementation is correct): ...
4
votes
1answer
286 views

Should my test set be balanced or imbalanced?

I have an imbalanced dataset (90% class 0 10% class 1), should I first split it in training and test set, then balance my training set (my test set would still be imbalanced), or could I randomly ...
4
votes
1answer
2k views

Diagnostic plots for lmer

I am trying to produce a glmm using the lme4 package in R. To validate my model I would like to produce some diagnostic plots ...
4
votes
1answer
155 views

ML / train-test-validate: What is allowed when?

As someone getting started in machine learning, I am trying to get my head around the rules / good practices to follow when building, testing and validating supervised ML models in order not to ...
4
votes
1answer
576 views

Compare modeled (fitted) paired data to actual data in forecasting problem (Excel sheet included)

In my endeavor to learn forecasting and improve my statistical knowledge, I've decided to forecast the population of a certain area. I've attached the excel file that I used. I know I'm using excel ...
4
votes
0answers
59 views

How to Validate a Monte Carlo Simulation

I have historical data of a production process, and I've being asked to build a simulation model to predict its performance in the future. Using the historical data, I've being able to obtain the ...
4
votes
0answers
86 views

optimism-corrected regression coefficients using Frank Harrell's method?

I used a regularized (LASSO) cox regression to estimate relapse times of patients and used Frank Harrell's bootstrapping method to obtain an optimism-corrected performance estimate of my model. I am ...
4
votes
0answers
1k views

Hold out sample vs. cross validation for time series, and how to perform in R

I think out-of-sample validation testing for accuracy is essential in initially judging what time-series forecasts to use. In any case, I've been doing some reading on the two most common methods, ...
4
votes
1answer
213 views

Minimal number of samples/conversions for statistical validity

We are measuring conversion rates (% of visitors who bought) on an e-commerce site. The test apply to a segment of visitors who meet specific criteria (for example people from a certain country). ...
3
votes
2answers
4k views

How do I validate my multiple linear regression model?

I have split my data into two parts. I have used my 80% data to build a multiple regression linear model. Now I want to test it using my rest 20% data. What tools on Minitab do I have to make this ...
3
votes
1answer
129 views

What is the difference between sensitivity analysis and model validation?

I read both wikipedia pages of sensitivity analysis and model validation (here, only linear regression validation) but I don't manage to find a way to separate these two terms. I have the impression ...
3
votes
2answers
4k views

Computing c-index for an external validation of a Cox PH model with R

First off, I'll state that I'm aware many questions get asked about the c-index. I've searched this site and others, and I haven't found an answer for my situation. I can successfully use ...
3
votes
1answer
39 views

Random Sample Validation

This is a real life scenario where I have just assumed responsibility for a new team The results of my team who perform legal contract abstractions are audited each month by our client using random ...