Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

35
votes
5answers
25k views

What is the difference between test set and validation set?

I found this confusing when I use the neural network toolbox in Matlab. It divided the raw data set into three parts: training set validation set test set I notice in many training or learning ...
24
votes
2answers
3k views

How to draw valid conclusions from “big data”?

"Big data" is everywhere in the media. Everybody says that "big data" is the big thing for 2012, e.g. KDNuggets poll on hot topics for 2012. However, I have deep concerns here. With big data, ...
20
votes
4answers
508 views

As a reviewer, can I justify requesting data and code be made available even if the journal does not?

As science must be reproducible, by definition, there is increasing recognition that data and code are an essential component of the reproduciblity, as discussed by the Yale Roundtable for data and ...
11
votes
1answer
101 views

When building a regression model using separate modeling/validation sets, is it appropriate to “recirculate” the validation data?

Suppose I've got an 80/20 split between modeling/validation observations. I've fit a model to the modeling data set, and I'm comfortable with the error that I'm seeing on the validation data set. ...
8
votes
2answers
2k views

What is the procedure for “bootstrap validation” (a.k.a. “resampling cross-validation”)?

"Bootstrap validation"/"resampling cross-validation" is new to me, but was discussed by the answer to this question. I gather it involves 2 types of data: the real data and simulated data, where a ...
6
votes
3answers
2k views

What is a consistency check?

I was asked such a question as "Did you do any consistency check in your daily work?" during a phone interview for a Biostatistician position. I don't know what to answer. Any information is ...
5
votes
1answer
257 views

Name of mean absolute error analogue to Brier score?

Yesterday's question Determine accuracy of model which estimates probability of event got me curious about probability scoring. The Brier score $$\frac{1}{N}\sum\limits _{i=1}^{N}(prediction_i - ...
5
votes
1answer
114 views

What do you do with your testing data?

Suppose you perform a parameter selection study in an environment with not-so-many data points. You divide the available data into training and testing set and you plan to validate the future model on ...
5
votes
2answers
5k views

How to make representative sample set from a large overall dataset?

What are the statistical techniques to create a sample set, which is representative of the entire population (with a known confidence level)? Also, How to validate, if the sample fits the overall ...
5
votes
2answers
169 views

How to do external validation of logistic regression models and perform model benchmarking

Quality assessment in trauma has for > 25 years been done with the US derived logistic regression model, the TRISS model. DV: survival/death and IVs: physiologic derangement (continuous), anatomic ...
5
votes
1answer
126 views

Validating a logistic regression for a specific $x$

I have a logistic regression model for 0/1 binary response data that is built from samples $(x_1,Y_1),\dots,(x_m,Y_m)$, where $x_1,\dots,x_m$ are, fixed, nonrandom, real values and $Y_1,\dots,Y_m$ are ...
4
votes
2answers
427 views

In logistic regression, does the lack of significance of the parameter estimates in a test sample indicate overfitting?

I am trying to build a logistic regression model where I have a dependent variable $y$ and independent variables $x_1$, $x_2$... $x_n$. $y$ can take only two values - 0 or 1. My original modelling ...
4
votes
1answer
2k views

Model validation after fitting a negative binomial GLM in R

Ok, I have searched and searched and just have no clue where to start. First, what I would like to do is produce a QQ-plot (or even a readable residual plot) to look at the fit of my model. I guess ...
4
votes
1answer
371 views

Best method to validate a multiply imputed Cox model with R?

This question is with regards to using a test data set to validate an imputed Cox model using R. With a non-imputed data set I would use val.surv() from ...
4
votes
1answer
1k views

Best practices for measuring and avoiding overfitting?

I am developing automated trading systems for the stock market. The big challenge has been overfitting. Can your recommend some resources describing methods for measuring and avoiding overfitting? I ...
4
votes
2answers
960 views

Variance explained of a mixed effects model in a new data set

I'm using the following function to calculate Edwards R^2 (formula 19 in Edwards et al. 2008) of a mixed effects model (I hope the implementation is correct): ...
4
votes
1answer
1k views

Diagnostic plots for lmer

I am trying to produce a glmm using the lme4 package in R. To validate my model I would like to produce some diagnostic plots ...
4
votes
1answer
347 views

Compare modeled (fitted) paired data to actual data in forecasting problem (Excel sheet included)

In my endeavor to learn forecasting and improve my statistical knowledge, I've decided to forecast the population of a certain area. I've attached the excel file that I used. I know I'm using excel ...
4
votes
0answers
126 views

Are world cup predictions testable?

As of today, dozens of soccer world cup predictions exist, some more complex, some more elegant, and most of them predict every nation's "chance" of winning a particular match/ the cup. As I am ...
4
votes
1answer
346 views

What is the intuition behind the variation of information (VI) metric for cluster validation?

For non-statisticians like me, it is very difficult to capture the idea of VI metric (variation of information) even after reading the relevant paper by Marina ...
4
votes
1answer
618 views

Validate cluster analysis in R

I am trying to validate hierarchical cluster analysis result following a paper by Guy Brock, et al. clValid: An R Package for Cluster Validation (pdf). Do I have to use all these methods? What are the ...
3
votes
2answers
1k views

How do I validate my multiple linear regression model?

I have split my data into two parts. I have used my 80% data to build a multiple regression linear model. Now I want to test it using my rest 20% data. What tools on Minitab do I have to make this ...
3
votes
2answers
328 views

k folds cross validation on a multi-class dataset

Cross validation is one of the most important tools because it gives us an honest assessment of the true accuracy of our system. In other words, the cross-validation process provides a much more ...
3
votes
1answer
263 views

Determining values of correction factor based on x bins in observed vs. actual data

I am trying to automate a problem I usually solve by hand. I have a sensor that collects data from the field. Every 6 months or so, I have to do a calibration on that sensor by collecting ...
3
votes
2answers
425 views

Logistic regression performs better on validation data

Recently I've been building a model using logistic regression. To my suprisise LIFT chart looks better on the validation data than on the training data, the same is with ROC. All variables in the ...
3
votes
1answer
680 views

How to choose training and test sets

I would like to propose a single model (decision tree), that is very variable, and validate it. I have choosen parameters after I had obtained good quality measures with a cross-validation. I could ...
3
votes
2answers
186 views

How to do external validation of regression models

Very basic question here, so bear with me... I have a data set with 241 patients with 16 variables plus diagnosis (malignant vs benign). There are 3 previously published logistic regression formulas ...
3
votes
1answer
54 views

Mixing User Data For Cross-Validation

I have data from 12 users, and want to perform cross-validation. Is it necessary that I create my training and test data from different users, or can all of the data be randomized, and then split into ...
3
votes
1answer
110 views

External model validation using new data for prediction: How large of a drop in $R^2$ is significant?

I need to validate a model using `external model validation' and I have a question relating to deciding when a drop in $R^2$ when compared to $R_{prediction}^2$ is significant. DISCLOSURE: This is ...
3
votes
1answer
106 views

Should my test set be balanced or imbalanced?

I have an imbalanced dataset (90% class 0 10% class 1), should I first split it in training and test set, then balance my training set (my test set would still be imbalanced), or could I randomly ...
3
votes
1answer
629 views

Using Adaboost for feature selection?

Is it okay to use Adaboost to do feature selection (selecting a subset of dimensions $S$ from a high-dimensional feature vector $V$)? I divided the samples into four non-overlapping sets: $A$ ...
3
votes
1answer
130 views

Statistical measures for data validation

I have to validate the data in one database by comparing it to data from a validated application system. The solution I came up with was (at least in theory) creating a statistical method that would ...
3
votes
1answer
163 views

Validation of a scale for a different population (CFA)

I'm currently adapting a coping questionnaire which has been widely used in sports to be used with a different population (musicians), so I need to address its reliability and validity with this new ...
3
votes
1answer
210 views

What resources/methods exist for testing/validation or evaluation of Statistical Methods

I'm interested in the process of testing or validating a particular implementation of a statistical method, and what datasets and/or published analysis exist that could be used to do this in practice. ...
3
votes
0answers
120 views

Internal validation via bootstrap: What ROC curve to present?

I am using the bootstrap approach for internal validation of a multivariate model built with either standard logistic regression OR elastic net. The procedure I use is as follows: 1) build model ...
3
votes
0answers
68 views

Validating statistical tests for value at risk and expected shortfall

I am trying to figure out if value-at-risk (VaR, a quantile) type tests could capture if expected shortfall (expectations above a quantile) point forecast generated from a type of model could be ...
3
votes
1answer
99 views

Validation of a questionnaire in a new population

I have 400 responses to a 20 item questionnaire which purports to measure an attitudinal constuct in medical students. The instrument was validated in the US for a single year of medical students and ...
3
votes
0answers
88 views

Validation of mixed-effect models

I want to use linear mixed effect model for a set of data. After using lme4 package and lmer() function and fitting model, I want to validate my model for other ...
3
votes
0answers
306 views

Logistic Regression Cost Function issue in Matlab

I'm trying to implement a logistic regression function in matlab. I calculated the theta values, linear regression cost function is converging and then I use those parameters in logistic regression ...
2
votes
1answer
87 views

Is a different CV arrangement the same as a validation set?

I have a smallish dataset ~ 1500 rows X 500 columns. I've been using a standard 5 fold CV setup where row 1 = CV set1, row2 = CV set2, ... row 6 = CV set1,etc. I'm at the point where I'm trying to ...
2
votes
2answers
87 views

Validate a medical test

I've never done something like this before, thus I do not even know where to start with calculating whatever I need. Following scenario: We measure a medical value (Glucose) with our device, and ...
2
votes
1answer
215 views

Validating a paper questionnaire in a web-based format

I would like to validate a commonly used questionnaire for internet-based administration. What statistical tests should I be using?
2
votes
2answers
776 views

What are acceptable validation or cross validation error rates?

Is there a commonly acceptable error rate for validation? As in, if the error rate is less than X %, then my machine learning method would be considered "successful". I'm looking for something ...
2
votes
4answers
520 views

Statistical tests to do data quality checks

I want to identify bad responses in the data we get from our vendors. To assess the quality of the data, what statistical tests can i apply to identify bad respondents? I performed Discriminant ...
2
votes
2answers
38 views

The size of the sample for split validation

At this moment I have a dataset with 4000 samples (50% positive and 50% negative). Normally I would do cross validation for this approach, however besides normal data mining techniques I am also ...
2
votes
2answers
1k views

Computing c-index for an external validation of a Cox PH model with R

First off, I'll state that I'm aware many questions get asked about the c-index. I've searched this site and others, and I haven't found an answer for my situation. I can successfully use ...
2
votes
1answer
78 views

Bayesian $\chi^2$ discrepancy

We define the $\chi^2$ discrepancy as $$T(y,\theta)=\sum_i\frac{(y_i-E(y_i|\theta))^2}{\mathrm{var}(y_i|\theta)}$$ (definition from Gelman). How do we apply this if $\theta$ is a random variable, and ...
2
votes
1answer
795 views

Weight variables for predictive model

I received a question today that I wasn't exactly sure how to answer. I have built a predictive model using a fairly basic logistic regression that works pretty well and fits our business needs. ...
2
votes
2answers
63 views

Split train//validation/test sets by time, is it correct?

Here's the scenario, slightly altered to a common one. Credit card fraud, payments for the last 12 months (a rolling window). Train with the data from the first 10 months, validate with data from the ...
2
votes
2answers
128 views

Sample size for segmentation validation

I've created a automatic algorithm liver segmentation and the next step is the method validation. I have 16 exams with a mean of 40 slices each and my doubt now is how many of these slices would have ...