Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

0
votes
0answers
10 views

How do I conduct model validation with glm models that have proportional response variable?

I would like to determine which environmental factors influence turtle nest survival. I ranked glm models with AIC using a step-wise procedure. The response variable is the proportion of eggs that ...
0
votes
0answers
11 views

How is final training stopped? [on hold]

How to know when to stop training the final model, which would be used for test set predictions? There is no validation set anymore... Update: The model is a convolutional neural network trained on ...
2
votes
1answer
178 views

Minimal number of samples/conversions for statistical validity

We are measuring conversion rates (% of visitors who bought) on an e-commerce site. The test apply to a segment of visitors who meet specific criteria (for example people from a certain country). ...
1
vote
1answer
439 views

Hold out sample vs. cross validation for time series, and how to perform in R

I think out-of-sample validation testing for accuracy is essential in initially judging what time-series forecasts to use. In any case, I've been doing some reading on the two most common methods, ...
3
votes
1answer
354 views

Prediction evaluation metric for panel/longitudinal data

I would like to evaluate several different models that provide predictions of behavior at a monthly level. The data is balanced, and $n=$100,000 and $T=$12. The outcome is attending a concert in a ...
4
votes
1answer
131 views

Within-group sum of squares of cluster

I have a multivariate dataset for which I have only a table including the cross-wise Euclidean distances between all points and a list giving the assignment of each point to one of several clusters. ...
1
vote
1answer
261 views

Optimism bias - estimates of prediction error

The book Elements of Statistical Learning (available in PDF online) discusses the optimisim bias (7.21, page 229). It states that the optimism bias is the difference between the training error and the ...
3
votes
1answer
169 views

Validation of a questionnaire in a new population

I have 400 responses to a 20 item questionnaire which purports to measure an attitudinal constuct in medical students. The instrument was validated in the US for a single year of medical students and ...
0
votes
0answers
15 views

Alternative to weighted kappa?

I want to compare modeled species occurrence with observed occurrence data. Using confusion matrix and weighed kappa seems to be a good option. In contrast to the normal kappa the weighted kappa takes ...
75
votes
7answers
55k views

What is the difference between test set and validation set?

I found this confusing when I use the neural network toolbox in Matlab. It divided the raw data set into three parts: training set validation set test set I notice in many training or learning ...
0
votes
0answers
23 views

Validating formative model - statistical analysis

Based upon literature I have developed a model which I want to validate. Data gathering will be done with the use of surveys. My knowledge of statistics, or more the lack thereof, has raised some ...
2
votes
2answers
56 views

Performing k-means clustering on a set of lines

I have a set of lines (y = numbers between 1 and 100, x= discrete) that I am trying to cluster to group similarly-shaped profiles. I have found that the profiles seem to cluster the cleanest when ...
1
vote
1answer
59 views

What is a good way to test a simple Recurrent Neural Network

I have coded up a simple real-value regression RNN in theano. What kind of dataset should I test it on? How should I go about testing it? My structure is: Univariate (for now) timeseries, ...
3
votes
2answers
898 views

How exactly to partition training-set for k-fold cross validation on multi-class dataset?

Cross validation is one of the most important tools because it gives us an honest assessment of the true accuracy of our system. In other words, the cross-validation process provides a much more ...
1
vote
2answers
67 views

Is validation set always necessary?

Lets say I did the following steps: Used some separate development set to select some features. Decided a priori to use only one learning algorithm (SVM) with only default parameter values. Trained ...
0
votes
0answers
44 views

K-fold cross validation for a glmer model with nested data

I'm working on a data set that contains a hierarchical data structure (i.e., GPS locations nested within individual animals). I'm using a generalized linear mixed effects modeling procedure (lme4 ...
2
votes
1answer
125 views

How do you use test data set after Cross-validation

In some lectures and tutorials they suggest to split your data in three parts: training, validation and test. But it is not clear how test data set should be used and how this approach is better than ...
0
votes
0answers
26 views

Clustering technique and validation for distance based on file compression

I have a distance matrix based on a normalized compression distance between files: $$ d(x, y) = \frac{ C(xy) - \min \{ C(x), C(y) \} } {\max \{ C(x), C(y) \}} $$ Here, $C(xy)$ is the concatenation ...
0
votes
0answers
8 views

Evaluate and report fit of a model on validation cohort(s)

I trained a random forest regression model M on a training set. I am interested in how well the model predicts the responses in 3 different validation sets. I am also interested in the characteristics ...
7
votes
2answers
130 views

How to make sure that a machine learning algorithm's implementation is correct?

Say there is a machine learning algorithm (e.g. classification) that is well known and implemented by the original creators of the algorithm. Yet all you have is the ability to use the algorithm but ...
2
votes
1answer
23 views

Why do clustering internal validation indices are decreasing with the number of clusters?

I get the same pattern for 3 different indices: Silhouettes, Dunn and Connectivity- as the number of clusters increases, the score decreases. I am using several clustering methods and several distance ...
0
votes
0answers
14 views

Which phrase is correct? “Out of sample” or “Hold out sample”?

Which one is correct phrase for data that model didn't see? For example i trained my model based on data-set from 200 to 2002. Now i want test my model using 2003 data-set. Which phrase is correct in ...
1
vote
1answer
22 views

Missing variable in the external validation cohort

I have developed a model on my training cohort and I intent to use an external dataset to validate the model. However, one of the predictors used in the model was not captured in the external ...
7
votes
5answers
947 views

How to calculate Area Under the Curve (AUC), or the c-statistic, by hand

I am interested in calculating area under the curve (AUC), or the c-statistic, by hand for a binary logistic regression model. For example, in the validation dataset, I have the true value for the ...
0
votes
0answers
15 views

How do I compare a new method of statistical sampling with an established one?

Lets say that: There exist 10 countries. I travel to each country and take a random subset of the population of that country and count how many of my subset are woman. I make a table of country to ...
2
votes
2answers
132 views

GLMM validation: weird qq & fit vs residual plots

I'm encountering problems with the results of a glmer model (lme4-package). Im trying to answer the question, whether a beaver ...
1
vote
1answer
28 views

K-fold cross validation for hierarchical data sets

I'm currently working on a data set that contains a hierarchical data structure (i.e., GPS locations nested within individual animals). Does anyone know how to write R code for this type of ...
1
vote
0answers
51 views

Sample size needed to validate classification/prediction model

Dose any rule of thumb exist (or possible calculation) regarding sample size needed to validate an binary classification model. We have developed this prediction model for a medical condition and ...
8
votes
2answers
712 views

What is the intuition behind the variation of information (VI) metric for cluster validation?

For non-statisticians like me, it is very difficult to capture the idea of VI metric (variation of information) even after reading the relevant paper by Marina ...
0
votes
1answer
43 views

Model validation using features (R)

I'm trying to create a classification model in R, at the moment i'm using 10 fold CV (using the train function of caret package) ...
0
votes
0answers
36 views

Testing model fit and running model validation plots for averaged models

I have carried out a model selection process and averaged the models within the top 2 AICcs (using MuMIn in R). I would like to test the model fit using both the chisq and validation/residual plots. ...
3
votes
1answer
139 views

Cross-validation for Comparing Clustering Techniques

I'm working on comparing multiple clustering algorithms to each other using the adjusted Rand index for a given dataset. We have a gold standard that we'd like to compare the obtained clustering ...
1
vote
0answers
211 views

Training/ Test Data with Time Series Model — Forecast with Training Model, or with Model based on Full Data?

Okay, I have a couple books on time series forecasting, but perhaps I need to read a couple more. Here's my question. You want to be able to validate a forecasting model. So you split the data into ...
2
votes
1answer
80 views

Can I apply factor analysis on multiple choice questions?

I am looking to validate a questionnaire and would like to know if I can use factor analysis on the multiple-choice questions (MCQ). Also, I have another section where I am asking about participants ...
1
vote
1answer
89 views

Creating a disease severity score/index

I am trying to design a numerical scale which would describe the severity of a certain disease (in this particular case anaphylaxis). I have a set of clinical symptoms and a database of patients who ...
0
votes
0answers
79 views

How to test/validate unlabeled data in association rules in R?

I produced association rules by using the arules package (apriori). I'm left with +/- 250 rules. I would like to test/validate the rules that I have, like answering the question: How do I know that ...
1
vote
0answers
319 views

Fitting Cox Regression / Proportional Hazard Model with x time interaction term in R

I am asking this in the context of wanting to diagnose for violation of proportional hazard assumption and its correction. (Schemper 1992) On p.179 of Hosmer, Lemeshow and May, it says that we can ...
0
votes
0answers
24 views

multiple linear model validation

Hello I am a beginner in R. I have a dataset of fish and environmental parameters and anthropogenic pressures from 11 lakes. I have modeled some fish parameters (like fish biomass) as function of ...
2
votes
1answer
66 views

Does the position at which maximum distance occurs in a KS test make a difference?

From my understanding of the KS test, fromt the CDF of two datasets, it measures the distance between the two distributions at various points and and compares the 'maximum distance' to a predefined ...
6
votes
4answers
231 views

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
1
vote
1answer
323 views

What is out of time validation in logistic regression model?

I understand out of sample validation very well. Can you explain what is out of time validation? Context A team in my organization has build a churn model for a teleco. Churn rate is 27%. The models ...
5
votes
2answers
1k views

Validate cluster analysis in R

I am trying to validate hierarchical cluster analysis result following a paper by Guy Brock, et al. clValid: An R Package for Cluster Validation (pdf). Do I have to use all these methods? What are the ...
1
vote
1answer
58 views

What is the name of this validation procedure?

I have a set of classified data. In order to test the precision of several algorithms- I split the data into train and test sets. For the test set I choose at random 30% of the data and the rest is ...
0
votes
0answers
35 views

Can the rcorrp.cens paired concordance test be done taking into account strata?

Followup to these discussions: Bootstrapping Hmisc::rcorrp.cens for paired concordance? Stratified concordance index (survival::survConcordance) Is there a way to perform a stratified paired ...
10
votes
3answers
1k views

Why isn't the holdout method (splitting data into training and testing) used in classical statistics?

In my classroom exposure to data mining, the holdout method was introduced as a way of assessing model performance. However, when I took my first class on linear models, this was not introduced as a ...
2
votes
2answers
134 views

rms validate on models with a predict function such as coxph and glmnet

I would like to use bootstrapping to evaluate models generated by coxph and glmnet. Would that be somehow possible with rms validate ? From the documentation it seems limited to rms functions (cph, ...
0
votes
0answers
93 views

Implementing Hopkins and Cox-Lewis index in R

I was trying to implement some clustering tendency tools in R, namely the Hopkin's index and the Cox–Lewis index. Here is the link at page 901 to show what they are This is what I managed to come up ...
1
vote
2answers
113 views

Repeatedly measuring accuracy against the hold out set

I have an iterative document classification task, corpus size = 300,000 documents. The labels are binary valued (yes/no). I wanted to know whether the following methodology is valid. The assumption is ...
0
votes
0answers
7 views

ANOVA for 4 groups 2 cohorts?

I have 2 differnt Cohorts from 2 differnt hospitals( unequal number within groups), Each is of 4 classes (differnt complexity) and I have mortaliy figures for each complexiy group. What is the best ...