Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

1
vote
1answer
30 views

What is the name of this validation procedure?

I have a set of classified data. In order to test the precision of several algorithms- I split the data into train and test sets. For the test set I choose at random 30% of the data and the rest is ...
0
votes
0answers
12 views

comparison behavior of time series [on hold]

I have three sets of time series data and I want to compare compare pro-all with two others. I want to validate our simulation output trend (c=0.1 and c=0.2) with real word trend (pro-all). I use ...
0
votes
0answers
14 views
+50

Can the rcorrp.cens paired concordance test be done taking into account strata?

Followup to these discussions: Bootstrapping Hmisc::rcorrp.cens for paired concordance? Stratified concordance index (survival::survConcordance) Is there a way to perform a stratified paired ...
8
votes
3answers
713 views

Why isn't the holdout method (splitting data into training and testing) used in classical statistics?

In my classroom exposure to data mining, the holdout method was introduced as a way of assessing model performance. However, when I took my first class on linear models, this was not introduced as a ...
3
votes
1answer
134 views

Validation of a questionnaire in a new population

I have 400 responses to a 20 item questionnaire which purports to measure an attitudinal constuct in medical students. The instrument was validated in the US for a single year of medical students and ...
1
vote
1answer
111 views

Optimism bias - estimates of prediction error

The book Elements of Statistical Learning (available in PDF online) discusses the optimisim bias (7.21, page 229). It states that the optimism bias is the difference between the training error and the ...
2
votes
1answer
267 views

Prediction evaluation metric for panel/longitudinal data

I would like to evaluate several different models that provide predictions of behavior at a monthly level. The data is balanced, and $n=$100,000 and $T=$12. The outcome is attending a concert in a ...
2
votes
2answers
93 views

rms validate on models with a predict function such as coxph and glmnet

I would like to use bootstrapping to evaluate models generated by coxph and glmnet. Would that be somehow possible with rms validate ? From the documentation it seems limited to rms functions (cph, ...
4
votes
1answer
487 views

What is the intuition behind the variation of information (VI) metric for cluster validation?

For non-statisticians like me, it is very difficult to capture the idea of VI metric (variation of information) even after reading the relevant paper by Marina ...
0
votes
0answers
22 views

Implementing Hopkins and Cox-Lewis index in R

I was trying to implement some clustering tendency tools in R, namely the Hopkin's index and the Cox–Lewis index. Here is the link at page 901 to show what they are This is what I managed to come up ...
1
vote
2answers
103 views

Repeatedly measuring accuracy against the hold out set

I have an iterative document classification task, corpus size = 300,000 documents. The labels are binary valued (yes/no). I wanted to know whether the following methodology is valid. The assumption is ...
2
votes
1answer
88 views

Cross-validation for Comparing Clustering Techniques

I'm working on comparing multiple clustering algorithms to each other using the adjusted Rand index for a given dataset. We have a gold standard that we'd like to compare the obtained clustering ...
0
votes
0answers
3 views

ANOVA for 4 groups 2 cohorts?

I have 2 differnt Cohorts from 2 differnt hospitals( unequal number within groups), Each is of 4 classes (differnt complexity) and I have mortaliy figures for each complexiy group. What is the best ...
5
votes
2answers
51 views

How to validate if a sample is independent and identically distributed

How can I check if the data is drawn i.i.d. from an unknown multivariate distribution?
54
votes
7answers
36k views

What is the difference between test set and validation set?

I found this confusing when I use the neural network toolbox in Matlab. It divided the raw data set into three parts: training set validation set test set I notice in many training or learning ...
1
vote
1answer
32 views

Variable selection and validation dataset

According to Hastie & Tibshirani, we shouldn't use validation datasets to do variable selection; otherwise, we will overestimate the model fit. However, it seems quite often to select variables ...
0
votes
0answers
11 views

Good mean shift clustering datasets

I am working on a modification of the Mean Shift algorithm and would like to validate my clustering results. I am struggling to find suitable clustering datasets to compare performance with. I'm ...
3
votes
0answers
365 views

Logistic Regression Cost Function issue in Matlab

I'm trying to implement a logistic regression function in matlab. I calculated the theta values, linear regression cost function is converging and then I use those parameters in logistic regression ...
0
votes
0answers
14 views

Can you measure the effectiveness of an estimator using only the recall?

Is it possible to measure the effectiveness of an estimator system Using only the recall measure? Suppose the we have a set of product P(P1, P2 ,... Pn) each product has its ideal reputation value ...
0
votes
0answers
14 views

Methods for ordered logistic regression validation?

We are building now a model using ordered logistic regression in R. I am trying to think one step ahead and am looking into possible approaches for validation. Currently we have like 800 obs ...
0
votes
0answers
6 views

The authenticity of the N-cut measure when the number of components in the data is high

I'm running a clustering task on unlabeled data, and assume we're validating our results by applying the Min-Cut measure as an internal validity index. Let's refer the normalized version of the ...
1
vote
1answer
82 views

Classification score for Random Forest

I'm learning about the Decision Tree and Random Forests. But there is something I don't really understand. I have a training set and a cross-validation set. I need to train different Random Forests, ...
0
votes
1answer
68 views

R times series — correct use of forecast() and accuracy() in forecast package

Cross-posting this from Stack Overflow, because it's a bit of a stats/ technology cross-over. I'm relatively new to R and the forecast package I believe authored by Rob Hyndman. I'm having trouble ...
1
vote
0answers
43 views

Training/ Test Data with Time Series Model — Forecast with Training Model, or with Model based on Full Data?

Okay, I have a couple books on time series forecasting, but perhaps I need to read a couple more. Here's my question. You want to be able to validate a forecasting model. So you split the data into ...
1
vote
0answers
87 views

Hold out sample vs. cross validation for time series, and how to perform in R

I think out-of-sample validation testing for accuracy is essential in initially judging what time-series forecasts to use. In any case, I've been doing some reading on the two most common methods, ...
4
votes
1answer
852 views

Validate cluster analysis in R

I am trying to validate hierarchical cluster analysis result following a paper by Guy Brock, et al. clValid: An R Package for Cluster Validation (pdf). Do I have to use all these methods? What are the ...
0
votes
2answers
38 views

Validating clustering results with labeled data

I am working on a clustering algorithm and would like to validate its performance against a well-known and used dataset: the KDD-CUP 99 dataset ...
2
votes
3answers
3k views

What is the meaning of orthogonal in validation testing?

I have heard the term "orthogonal * validation" used recently. It was used in the context of experimental platform testing. What does this mean? I cannot find anything on it in literature or ...
0
votes
0answers
42 views

Quantifying the predictive ability of a model developed from a huge data set? (variation of bootstrapping?)

I have a statistical model with around 20 predictor variables, built on 90% of a dataset consisting of over 600k observations. The original developer held out 10% of the original dataset for the ...
0
votes
2answers
26 views

Accepted Method for Selecting a Validation Set in Python

Is there an accepted method for separating out a validation set in python? In R I would use the sample function. I have 4000 training instances as json and I want ...
0
votes
1answer
75 views

Ignore strata in external validation of stratified Cox prop hazards model?

I've fit a stratified Cox proportional hazards model to some survival data, where I've stratified by a potential confounder which is the batch the data comes from (there are three batches). Now, I'd ...
3
votes
1answer
780 views

Using Adaboost for feature selection?

Is it okay to use Adaboost to do feature selection (selecting a subset of dimensions $S$ from a high-dimensional feature vector $V$)? I divided the samples into four non-overlapping sets: $A$ ...
0
votes
0answers
186 views

How to interpret the results of bootstrapping and Monte Carlo simulation utilised to test lasso logistic regression results?

My situation: sample size: 116 binary outcome (32 events) number predictors: 42 (both continuous and categorical) predictors did not come from the top of my head; their choice was based on the ...
0
votes
0answers
98 views

Fitting Cox Regression / Proportional Hazard Model with x time interaction term in R

I am asking this in the context of wanting to diagnose for violation of proportional hazard assumption and its correction. (Schemper 1992) On p.179 of Hosmer, Lemeshow and May, it says that we can ...
1
vote
3answers
240 views

Performance drop between training and validation datasets

I have been using R's GBM (Gradient Boosting Machine) package for several months. I typically split my data into three partitions: Training, Validation, and Testing. I use the validation data set to ...
3
votes
1answer
81 views

What does a negative Somers' D say about model discriminative power?

I developed a Cox proportional hazard regression model in R. Then I tried using validate in Professor Harrell's rms package to ...
1
vote
1answer
44 views

10-fold cross-validation (high variation)

I am using 10-fold validation method to validate my model. I am using CART model and my sample size $\approx$ 50. Features $\approx$ 9. The 10-fold validated accuracy (averages) is about 76%. However, ...
3
votes
2answers
960 views

Validity of pseudo-panel data constructed from repeated cross sectional data as a panel data

I am looking at the repeated cross-sectional data from federal reserves, which has both panel data and repeated cross sectional data at different time-points,e.g. 2007-2009 is a panel while 2010 is a ...
0
votes
0answers
17 views

cross-validate hierarchical model for binomial data that is often sparse

I have binomial data (e.g, 130 successes in 4000 trials). In many, if not most, cells of interest, there were few trials and thereby few successes (e.g., 0 successes in 35 trials, 1 successes in 18 ...
0
votes
0answers
23 views

A way to check the accuracy of a Markov chain?

I posted the same question on MSE since I was not really sure whether to post it here first or not. Anyway since I still did not get any answers I will be posting this here hoping for some help. Say ...
0
votes
0answers
21 views

How to measure classification accuracy based on presence only data?

I have a binomial GLMM which I calibrated with data on recreational visits (presence) compared with random controls where no visits were recorded (absence). I generated the controls myself, whereas ...
0
votes
0answers
10 views

Select final model after many Dev-Val splits

We have a dataset that is quite heterogeneous, and we established that different Dev-Val splits affect the outcome quite bit, even after using stratified sampling. So what we did write a loop around ...
1
vote
1answer
32 views

How to validate a lognormal random walk for time series data

I am currently working on a project where I need to simulate the prices of a set of $D$ substitutable commodities over time. I was hoping to do this using the following $D$-dimensional lognormal ...
0
votes
1answer
298 views

How to use k-fold cross validation in naive bayes classifier?

I'm trying to classify text using naive bayes classifier, and also want to use k-fold cross validation to validate the result of classification. But I'm still confused how to use the k-fold cross ...
0
votes
1answer
15 views

what “non-continuous nature of scores” means

1) Can someone explain me the idea of this sentence: "It is noted that nonparametric tests were employed because of the non-continuous nature of the scale scores" (this is related to scale validity ...
0
votes
0answers
14 views

Validation of the gene-signature predictor reconstructed using Cox PH

I reconstructed a prognostic gene model including four genes using univariate and then multivariate Cox PH model. The model is used for predicting prognosis in patients with a specific cancer ...
0
votes
1answer
84 views

Validate a Markov Chain with few states (model switching)

Suppose I have a five state Markov chain. The states are observable (in fact I defined them, they are outcomes of an classification algorithm). So I have a long time series, see the first picture. ...
0
votes
1answer
31 views

How can one evaluate Incremental Clustering Algorithms, in particular the goodness of the clusters formed?

I have been studying an incremental clustering algorithm for a large set of data that exhibit an inherent dynamic behavior (that is new data can get added over time and some older data may get deleted ...
0
votes
0answers
9 views

What is the minimal amount of sample runs required for doing a minimum detection limit?

I have to set up an example of how to do an MDL, and the professor wants to know how many times I'd run the samples. My previous class used as little as 8, but I'm not sure if thats correct or if I ...
1
vote
1answer
165 views

How do you validate your machine learning models?

I am wondering what approaches are commonly used for validating machine learning models designed for classification or prediction tasks: Approaches that am using at the moment: Using truth-sets: - ...