Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

128
votes
9answers
117k views

What is the difference between test set and validation set?

I found this confusing when I use the neural network toolbox in Matlab. It divided the raw data set into three parts: training set validation set test set I notice in many training or learning ...
30
votes
2answers
4k views

How to draw valid conclusions from “big data”?

"Big data" is everywhere in the media. Everybody says that "big data" is the big thing for 2012, e.g. KDNuggets poll on hot topics for 2012. However, I have deep concerns here. With big data, ...
22
votes
4answers
722 views

As a reviewer, can I justify requesting data and code be made available even if the journal does not?

As science must be reproducible, by definition, there is increasing recognition that data and code are an essential component of the reproduciblity, as discussed by the Yale Roundtable for data and ...
20
votes
5answers
9k views

Hold-out validation vs. cross-validation

To me, it seems that hold-out validation is useless. That is, splitting the original dataset into two-parts (training and testing) and using the testing score as a generalization measure, is somewhat ...
11
votes
3answers
1k views

Why isn't the holdout method (splitting data into training and testing) used in classical statistics?

In my classroom exposure to data mining, the holdout method was introduced as a way of assessing model performance. However, when I took my first class on linear models, this was not introduced as a ...
11
votes
1answer
131 views

When building a regression model using separate modeling/validation sets, is it appropriate to “recirculate” the validation data?

Suppose I've got an 80/20 split between modeling/validation observations. I've fit a model to the modeling data set, and I'm comfortable with the error that I'm seeing on the validation data set. ...
10
votes
2answers
4k views

What is the procedure for “bootstrap validation” (a.k.a. “resampling cross-validation”)?

"Bootstrap validation"/"resampling cross-validation" is new to me, but was discussed by the answer to this question. I gather it involves 2 types of data: the real data and simulated data, where a ...
9
votes
3answers
5k views

What is a consistency check?

I was asked such a question as "Did you do any consistency check in your daily work?" during a phone interview for a Biostatistician position. I don't know what to answer. Any information is ...
9
votes
4answers
977 views

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
8
votes
2answers
353 views

Should final (production ready) model be trained on complete data or just on training set?

Suppose I trained several models on training set, choose best one using cross validation set and measured performance on test set. So now I have one final best model. Should I retrain it on my all ...
8
votes
1answer
2k views

What is the intuition behind the variation of information (VI) metric for cluster validation?

For non-statisticians like me, it is very difficult to capture the idea of VI metric (variation of information) even after reading the relevant paper by Marina ...
7
votes
2answers
195 views

How to make sure that a machine learning algorithm's implementation is correct?

Say there is a machine learning algorithm (e.g. classification) that is well known and implemented by the original creators of the algorithm. Yet all you have is the ability to use the algorithm but ...
6
votes
1answer
515 views

Name of mean absolute error analogue to Brier score?

Yesterday's question Determine accuracy of model which estimates probability of event got me curious about probability scoring. The Brier score $$\frac{1}{N}\sum\limits _{i=1}^{N}(prediction_i - ...
6
votes
1answer
2k views

Best practices for measuring and avoiding overfitting?

I am developing automated trading systems for the stock market. The big challenge has been overfitting. Can your recommend some resources describing methods for measuring and avoiding overfitting? I ...
6
votes
1answer
92 views

How to account for case weights when generating folds for K-fold CV?

I am currently working on a binary classification problem where each point in the dataset is paired with a case weight. Specifically, each record $(w_i, x_i, y_i)$ represents a distinct type of ...
6
votes
1answer
1k views

Within-group sum of squares of cluster

I have a multivariate dataset for which I have only a table including the cross-wise Euclidean distances between all points and a list giving the assignment of each point to one of several clusters. ...
6
votes
2answers
641 views

Internal validation via bootstrap: What ROC curve to present?

I am using the bootstrap approach for internal validation of a multivariate model built with either standard logistic regression OR elastic net. The procedure I use is as follows: 1) build model ...
5
votes
1answer
83 views

What can be inferred from this residual plot?

This is with reference to http://analyticspro.org/2016/03/05/r-tutorial-residual-analysis-for-regression/. For Residual plot (b), where the residuals are increasing linearly (with respect to the ...
5
votes
2answers
348 views

How to validate if a sample is independent and identically distributed

How can I check if the data is drawn i.i.d. from an unknown multivariate distribution?
5
votes
1answer
124 views

What do you do with your testing data?

Suppose you perform a parameter selection study in an environment with not-so-many data points. You divide the available data into training and testing set and you plan to validate the future model on ...
5
votes
2answers
10k views

How to make representative sample set from a large overall dataset?

What are the statistical techniques to create a sample set, which is representative of the entire population (with a known confidence level)? Also, How to validate, if the sample fits the overall ...
5
votes
2answers
694 views

How to do external validation of logistic regression models and perform model benchmarking

Quality assessment in trauma has for > 25 years been done with the US derived logistic regression model, the TRISS model. DV: survival/death and IVs: physiologic derangement (continuous), anatomic ...
5
votes
2answers
2k views

Validate cluster analysis in R

I am trying to validate hierarchical cluster analysis result following a paper by Guy Brock, et al. clValid: An R Package for Cluster Validation (pdf). Do I have to use all these methods? What are the ...
5
votes
1answer
136 views

Validating a logistic regression for a specific $x$

I have a logistic regression model for 0/1 binary response data that is built from samples $(x_1,Y_1),\dots,(x_m,Y_m)$, where $x_1,\dots,x_m$ are, fixed, nonrandom, real values and $Y_1,\dots,Y_m$ are ...
5
votes
0answers
152 views

Are world cup predictions testable?

As of today, dozens of soccer world cup predictions exist, some more complex, some more elegant, and most of them predict every nation's "chance" of winning a particular match/ the cup. As I am ...
5
votes
1answer
481 views

Prediction evaluation metric for panel/longitudinal data

I would like to evaluate several different models that provide predictions of behavior at a monthly level. The data is balanced, and $n=$100,000 and $T=$12. The outcome is attending a concert in a ...
4
votes
2answers
1k views

How exactly to partition training-set for k-fold cross validation on multi-class dataset?

Cross validation is one of the most important tools because it gives us an honest assessment of the true accuracy of our system. In other words, the cross-validation process provides a much more ...
4
votes
2answers
707 views

In logistic regression, does the lack of significance of the parameter estimates in a test sample indicate overfitting?

I am trying to build a logistic regression model where I have a dependent variable $y$ and independent variables $x_1$, $x_2$... $x_n$. $y$ can take only two values - 0 or 1. My original modelling ...
4
votes
2answers
453 views

What is a good way to test a simple Recurrent Neural Network

I have coded up a simple real-value regression RNN in theano. What kind of dataset should I test it on? How should I go about testing it? My structure is: Univariate (for now) timeseries, ...
4
votes
2answers
409 views

How to do external validation of regression models

Very basic question here, so bear with me... I have a data set with 241 patients with 16 variables plus diagnosis (malignant vs benign). There are 3 previously published logistic regression formulas ...
4
votes
1answer
4k views

Model validation after fitting a negative binomial GLM in R

Ok, I have searched and searched and just have no clue where to start. First, what I would like to do is produce a QQ-plot (or even a readable residual plot) to look at the fit of my model. I guess ...
4
votes
1answer
1k views

Using Adaboost for feature selection?

Is it okay to use Adaboost to do feature selection (selecting a subset of dimensions $S$ from a high-dimensional feature vector $V$)? I divided the samples into four non-overlapping sets: $A$ ...
4
votes
1answer
798 views

Best method to validate a multiply imputed Cox model with R?

This question is with regards to using a test data set to validate an imputed Cox model using R. With a non-imputed data set I would use val.surv() from ...
4
votes
2answers
1k views

Variance explained of a mixed effects model in a new data set

I'm using the following function to calculate Edwards R^2 (formula 19 in Edwards et al. 2008) of a mixed effects model (I hope the implementation is correct): ...
4
votes
1answer
257 views

Should my test set be balanced or imbalanced?

I have an imbalanced dataset (90% class 0 10% class 1), should I first split it in training and test set, then balance my training set (my test set would still be imbalanced), or could I randomly ...
4
votes
1answer
2k views

Diagnostic plots for lmer

I am trying to produce a glmm using the lme4 package in R. To validate my model I would like to produce some diagnostic plots ...
4
votes
1answer
142 views

ML / train-test-validate: What is allowed when?

As someone getting started in machine learning, I am trying to get my head around the rules / good practices to follow when building, testing and validating supervised ML models in order not to ...
4
votes
1answer
563 views

Compare modeled (fitted) paired data to actual data in forecasting problem (Excel sheet included)

In my endeavor to learn forecasting and improve my statistical knowledge, I've decided to forecast the population of a certain area. I've attached the excel file that I used. I know I'm using excel ...
4
votes
0answers
1k views

Hold out sample vs. cross validation for time series, and how to perform in R

I think out-of-sample validation testing for accuracy is essential in initially judging what time-series forecasts to use. In any case, I've been doing some reading on the two most common methods, ...
3
votes
2answers
4k views

How do I validate my multiple linear regression model?

I have split my data into two parts. I have used my 80% data to build a multiple regression linear model. Now I want to test it using my rest 20% data. What tools on Minitab do I have to make this ...
3
votes
1answer
92 views

What is the difference between sensitivity analysis and model validation?

I read both wikipedia pages of sensitivity analysis and model validation (here, only linear regression validation) but I don't manage to find a way to separate these two terms. I have the impression ...
3
votes
1answer
38 views

Random Sample Validation

This is a real life scenario where I have just assumed responsibility for a new team The results of my team who perform legal contract abstractions are audited each month by our client using random ...
3
votes
1answer
3k views

Classification score for Random Forest

I'm learning about the Decision Tree and Random Forests. But there is something I don't really understand. I have a training set and a cross-validation set. I need to train different Random Forests, ...
3
votes
1answer
495 views

Determining values of correction factor based on x bins in observed vs. actual data

I am trying to automate a problem I usually solve by hand. I have a sensor that collects data from the field. Every 6 months or so, I have to do a calibration on that sensor by collecting ...
3
votes
2answers
4k views

Computing c-index for an external validation of a Cox PH model with R

First off, I'll state that I'm aware many questions get asked about the c-index. I've searched this site and others, and I haven't found an answer for my situation. I can successfully use ...
3
votes
2answers
596 views

Logistic regression performs better on validation data

Recently I've been building a model using logistic regression. To my suprisise LIFT chart looks better on the validation data than on the training data, the same is with ROC. All variables in the ...
3
votes
1answer
94 views

Does the position at which maximum distance occurs in a KS test make a difference?

From my understanding of the KS test, fromt the CDF of two datasets, it measures the distance between the two distributions at various points and and compares the 'maximum distance' to a predefined ...
3
votes
1answer
338 views

What does a negative Somers' D say about model discriminative power?

I developed a Cox proportional hazard regression model in R. Then I tried using validate in Professor Harrell's rms package to ...
3
votes
2answers
1k views

Validity of pseudo-panel data constructed from repeated cross sectional data as a panel data

I am looking at the repeated cross-sectional data from federal reserves, which has both panel data and repeated cross sectional data at different time-points,e.g. 2007-2009 is a panel while 2010 is a ...
3
votes
4answers
9k views

What is the meaning of orthogonal in validation testing?

I have heard the term "orthogonal * validation" used recently. It was used in the context of experimental platform testing. What does this mean? I cannot find anything on it in literature or ...