Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

0
votes
1answer
18 views

Calibration and validation of an accelerated failure time model on new data

I have an existing AFT (accelerated failure time) model which I'm using on a new dataset, with the obvious intent of testing whether the model predicts the new data well. A first step is to look at ...
0
votes
0answers
14 views

validation set with ranking variables

i'm working on an approach of feature selection with SVM model and i have some questions about validation , training and test sets. the idea is to rank variables in decreasing order of relevance ...
0
votes
0answers
13 views

Why use clustering + external measure for validation, instead of supervised learning?

Lets say I have set of genes in individuals and I will use clustering to find groups of individuals with similar genes and then I will check how these groups express itself in form of some disease. I ...
0
votes
0answers
11 views

How do I conduct model validation with glm models that have proportional response variable?

I would like to determine which environmental factors influence turtle nest survival. I ranked glm models with AIC using a step-wise procedure. The response variable is the proportion of eggs that ...
0
votes
0answers
15 views

Alternative to weighted kappa?

I want to compare modeled species occurrence with observed occurrence data. Using confusion matrix and weighed kappa seems to be a good option. In contrast to the normal kappa the weighted kappa takes ...
0
votes
0answers
26 views

Validating formative model - statistical analysis

Based upon literature I have developed a model which I want to validate. Data gathering will be done with the use of surveys. My knowledge of statistics, or more the lack thereof, has raised some ...
2
votes
2answers
63 views

Performing k-means clustering on a set of lines

I have a set of lines (y = numbers between 1 and 100, x= discrete) that I am trying to cluster to group similarly-shaped profiles. I have found that the profiles seem to cluster the cleanest when ...
1
vote
1answer
69 views

What is a good way to test a simple Recurrent Neural Network

I have coded up a simple real-value regression RNN in theano. What kind of dataset should I test it on? How should I go about testing it? My structure is: Univariate (for now) timeseries, ...
1
vote
2answers
81 views

Is validation set always necessary?

Lets say I did the following steps: Used some separate development set to select some features. Decided a priori to use only one learning algorithm (SVM) with only default parameter values. Trained ...
2
votes
0answers
53 views

Using simulated data to check when patterns in GLMM residual plots are acceptable

I have run the following Poisson GLMM: ...
0
votes
0answers
49 views

K-fold cross validation for a glmer model with nested data

I'm working on a data set that contains a hierarchical data structure (i.e., GPS locations nested within individual animals). I'm using a generalized linear mixed effects modeling procedure (lme4 ...
2
votes
1answer
161 views

How do you use test data set after Cross-validation

In some lectures and tutorials they suggest to split your data in three parts: training, validation and test. But it is not clear how test data set should be used and how this approach is better than ...
0
votes
0answers
26 views

Clustering technique and validation for distance based on file compression

I have a distance matrix based on a normalized compression distance between files: $$ d(x, y) = \frac{ C(xy) - \min \{ C(x), C(y) \} } {\max \{ C(x), C(y) \}} $$ Here, $C(xy)$ is the concatenation ...
0
votes
0answers
10 views

Evaluate and report fit of a model on validation cohort(s)

I trained a random forest regression model M on a training set. I am interested in how well the model predicts the responses in 3 different validation sets. I am also interested in the characteristics ...
7
votes
2answers
135 views

How to make sure that a machine learning algorithm's implementation is correct?

Say there is a machine learning algorithm (e.g. classification) that is well known and implemented by the original creators of the algorithm. Yet all you have is the ability to use the algorithm but ...
0
votes
0answers
14 views

Which phrase is correct? “Out of sample” or “Hold out sample”?

Which one is correct phrase for data that model didn't see? For example i trained my model based on data-set from 200 to 2002. Now i want test my model using 2003 data-set. Which phrase is correct in ...
2
votes
1answer
25 views

Why do clustering internal validation indices are decreasing with the number of clusters?

I get the same pattern for 3 different indices: Silhouettes, Dunn and Connectivity- as the number of clusters increases, the score decreases. I am using several clustering methods and several distance ...
1
vote
1answer
23 views

Missing variable in the external validation cohort

I have developed a model on my training cohort and I intent to use an external dataset to validate the model. However, one of the predictors used in the model was not captured in the external ...
0
votes
0answers
15 views

How do I compare a new method of statistical sampling with an established one?

Lets say that: There exist 10 countries. I travel to each country and take a random subset of the population of that country and count how many of my subset are woman. I make a table of country to ...
7
votes
5answers
1k views

How to calculate Area Under the Curve (AUC), or the c-statistic, by hand

I am interested in calculating area under the curve (AUC), or the c-statistic, by hand for a binary logistic regression model. For example, in the validation dataset, I have the true value for the ...
2
votes
2answers
151 views

GLMM validation: weird qq & fit vs residual plots

I'm encountering problems with the results of a glmer model (lme4-package). Im trying to answer the question, whether a beaver ...
1
vote
1answer
32 views

K-fold cross validation for hierarchical data sets

I'm currently working on a data set that contains a hierarchical data structure (i.e., GPS locations nested within individual animals). Does anyone know how to write R code for this type of ...
1
vote
0answers
57 views

Sample size needed to validate classification/prediction model

Dose any rule of thumb exist (or possible calculation) regarding sample size needed to validate an binary classification model. We have developed this prediction model for a medical condition and ...
0
votes
1answer
43 views

Model validation using features (R)

I'm trying to create a classification model in R, at the moment i'm using 10 fold CV (using the train function of caret package) ...
0
votes
0answers
44 views

Testing model fit and running model validation plots for averaged models

I have carried out a model selection process and averaged the models within the top 2 AICcs (using MuMIn in R). I would like to test the model fit using both the chisq and validation/residual plots. ...
5
votes
1answer
159 views

Within-group sum of squares of cluster

I have a multivariate dataset for which I have only a table including the cross-wise Euclidean distances between all points and a list giving the assignment of each point to one of several clusters. ...
2
votes
1answer
86 views

Can I apply factor analysis on multiple choice questions?

I am looking to validate a questionnaire and would like to know if I can use factor analysis on the multiple-choice questions (MCQ). Also, I have another section where I am asking about participants ...
1
vote
1answer
91 views

Creating a disease severity score/index

I am trying to design a numerical scale which would describe the severity of a certain disease (in this particular case anaphylaxis). I have a set of clinical symptoms and a database of patients who ...
0
votes
0answers
86 views

How to test/validate unlabeled data in association rules in R?

I produced association rules by using the arules package (apriori). I'm left with +/- 250 rules. I would like to test/validate the rules that I have, like answering the question: How do I know that ...
0
votes
0answers
24 views

multiple linear model validation

Hello I am a beginner in R. I have a dataset of fish and environmental parameters and anthropogenic pressures from 11 lakes. I have modeled some fish parameters (like fish biomass) as function of ...
3
votes
1answer
69 views

Does the position at which maximum distance occurs in a KS test make a difference?

From my understanding of the KS test, fromt the CDF of two datasets, it measures the distance between the two distributions at various points and and compares the 'maximum distance' to a predefined ...
1
vote
1answer
422 views

What is out of time validation in logistic regression model?

I understand out of sample validation very well. Can you explain what is out of time validation? Context A team in my organization has build a churn model for a teleco. Churn rate is 27%. The models ...
6
votes
4answers
267 views

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
1
vote
1answer
59 views

What is the name of this validation procedure?

I have a set of classified data. In order to test the precision of several algorithms- I split the data into train and test sets. For the test set I choose at random 30% of the data and the rest is ...
10
votes
3answers
1k views

Why isn't the holdout method (splitting data into training and testing) used in classical statistics?

In my classroom exposure to data mining, the holdout method was introduced as a way of assessing model performance. However, when I took my first class on linear models, this was not introduced as a ...
0
votes
0answers
35 views

Can the rcorrp.cens paired concordance test be done taking into account strata?

Followup to these discussions: Bootstrapping Hmisc::rcorrp.cens for paired concordance? Stratified concordance index (survival::survConcordance) Is there a way to perform a stratified paired ...
0
votes
0answers
106 views

Implementing Hopkins and Cox-Lewis index in R

I was trying to implement some clustering tendency tools in R, namely the Hopkin's index and the Cox–Lewis index. Here is the link at page 901 to show what they are This is what I managed to come up ...
0
votes
0answers
7 views

ANOVA for 4 groups 2 cohorts?

I have 2 differnt Cohorts from 2 differnt hospitals( unequal number within groups), Each is of 4 classes (differnt complexity) and I have mortaliy figures for each complexiy group. What is the best ...
5
votes
2answers
132 views

How to validate if a sample is independent and identically distributed

How can I check if the data is drawn i.i.d. from an unknown multivariate distribution?
0
votes
0answers
20 views

Good mean shift clustering datasets

I am working on a modification of the Mean Shift algorithm and would like to validate my clustering results. I am struggling to find suitable clustering datasets to compare performance with. I'm ...
0
votes
0answers
18 views

Can you measure the effectiveness of an estimator using only the recall?

Is it possible to measure the effectiveness of an estimator system Using only the recall measure? Suppose the we have a set of product P(P1, P2 ,... Pn) each product has its ideal reputation value ...
0
votes
0answers
28 views

Methods for ordered logistic regression validation?

We are building now a model using ordered logistic regression in R. I am trying to think one step ahead and am looking into possible approaches for validation. Currently we have like 800 obs ...
0
votes
0answers
8 views

The authenticity of the N-cut measure when the number of components in the data is high

I'm running a clustering task on unlabeled data, and assume we're validating our results by applying the Min-Cut measure as an internal validity index. Let's refer the normalized version of the ...
1
vote
1answer
801 views

Classification score for Random Forest

I'm learning about the Decision Tree and Random Forests. But there is something I don't really understand. I have a training set and a cross-validation set. I need to train different Random Forests, ...
1
vote
0answers
239 views

Training/ Test Data with Time Series Model — Forecast with Training Model, or with Model based on Full Data?

Okay, I have a couple books on time series forecasting, but perhaps I need to read a couple more. Here's my question. You want to be able to validate a forecasting model. So you split the data into ...
0
votes
1answer
391 views

R times series — correct use of forecast() and accuracy() in forecast package

Cross-posting this from Stack Overflow, because it's a bit of a stats/ technology cross-over. I'm relatively new to R and the forecast package I believe authored by Rob Hyndman. I'm having trouble ...
1
vote
1answer
490 views

Hold out sample vs. cross validation for time series, and how to perform in R

I think out-of-sample validation testing for accuracy is essential in initially judging what time-series forecasts to use. In any case, I've been doing some reading on the two most common methods, ...
1
vote
1answer
66 views

Variable selection and validation dataset

According to Hastie & Tibshirani, we shouldn't use validation datasets to do variable selection; otherwise, we will overestimate the model fit. However, it seems quite often to select variables ...
0
votes
2answers
71 views

Validating clustering results with labeled data

I am working on a clustering algorithm and would like to validate its performance against a well-known and used dataset: the KDD-CUP 99 dataset ...
0
votes
0answers
52 views

Quantifying the predictive ability of a model developed from a huge data set? (variation of bootstrapping?)

I have a statistical model with around 20 predictor variables, built on 90% of a dataset consisting of over 600k observations. The original developer held out 10% of the original dataset for the ...