Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

0
votes
0answers
15 views

Evaluate and report fit of a model on validation cohort(s)

I trained a random forest regression model M on a training set. I am interested in how well the model predicts the responses in 3 different validation sets. I am also interested in the characteristics ...
7
votes
2answers
165 views

How to make sure that a machine learning algorithm's implementation is correct?

Say there is a machine learning algorithm (e.g. classification) that is well known and implemented by the original creators of the algorithm. Yet all you have is the ability to use the algorithm but ...
0
votes
0answers
24 views

Which phrase is correct? “Out of sample” or “Hold out sample”?

Which one is correct phrase for data that model didn't see? For example i trained my model based on data-set from 200 to 2002. Now i want test my model using 2003 data-set. Which phrase is correct in ...
2
votes
1answer
50 views

Why do clustering internal validation indices are decreasing with the number of clusters?

I get the same pattern for 3 different indices: Silhouettes, Dunn and Connectivity- as the number of clusters increases, the score decreases. I am using several clustering methods and several distance ...
1
vote
1answer
34 views

Missing variable in the external validation cohort

I have developed a model on my training cohort and I intent to use an external dataset to validate the model. However, one of the predictors used in the model was not captured in the external ...
0
votes
0answers
15 views

How do I compare a new method of statistical sampling with an established one?

Lets say that: There exist 10 countries. I travel to each country and take a random subset of the population of that country and count how many of my subset are woman. I make a table of country to ...
14
votes
5answers
10k views

How to calculate Area Under the Curve (AUC), or the c-statistic, by hand

I am interested in calculating area under the curve (AUC), or the c-statistic, by hand for a binary logistic regression model. For example, in the validation dataset, I have the true value for the ...
2
votes
2answers
325 views

GLMM validation: weird qq & fit vs residual plots

I'm encountering problems with the results of a glmer model (lme4-package). Im trying to answer the question, whether a beaver ...
1
vote
1answer
49 views

K-fold cross validation for hierarchical data sets

I'm currently working on a data set that contains a hierarchical data structure (i.e., GPS locations nested within individual animals). Does anyone know how to write R code for this type of ...
1
vote
0answers
93 views

Sample size needed to validate classification/prediction model

Dose any rule of thumb exist (or possible calculation) regarding sample size needed to validate an binary classification model. We have developed this prediction model for a medical condition and ...
0
votes
1answer
48 views

Model validation using features (R)

I'm trying to create a classification model in R, at the moment i'm using 10 fold CV (using the train function of caret package) ...
0
votes
0answers
80 views

Testing model fit and running model validation plots for averaged models

I have carried out a model selection process and averaged the models within the top 2 AICcs (using MuMIn in R). I would like to test the model fit using both the chisq and validation/residual plots. ...
5
votes
1answer
742 views

Within-group sum of squares of cluster

I have a multivariate dataset for which I have only a table including the cross-wise Euclidean distances between all points and a list giving the assignment of each point to one of several clusters. ...
2
votes
1answer
160 views

Can I apply factor analysis on multiple choice questions?

I am looking to validate a questionnaire and would like to know if I can use factor analysis on the multiple-choice questions (MCQ). Also, I have another section where I am asking about participants ...
1
vote
1answer
135 views

Creating a disease severity score/index

I am trying to design a numerical scale which would describe the severity of a certain disease (in this particular case anaphylaxis). I have a set of clinical symptoms and a database of patients who ...
0
votes
0answers
159 views

How to test/validate unlabeled data in association rules in R?

I produced association rules by using the arules package (apriori). I'm left with +/- 250 rules. I would like to test/validate the rules that I have, like answering the question: How do I know that ...
0
votes
0answers
32 views

multiple linear model validation

Hello I am a beginner in R. I have a dataset of fish and environmental parameters and anthropogenic pressures from 11 lakes. I have modeled some fish parameters (like fish biomass) as function of ...
3
votes
1answer
86 views

Does the position at which maximum distance occurs in a KS test make a difference?

From my understanding of the KS test, fromt the CDF of two datasets, it measures the distance between the two distributions at various points and and compares the 'maximum distance' to a predefined ...
1
vote
1answer
1k views

What is out of time validation in logistic regression model?

I understand out of sample validation very well. Can you explain what is out of time validation? Context A team in my organization has build a churn model for a teleco. Churn rate is 27%. The models ...
8
votes
4answers
676 views

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
1
vote
1answer
62 views

What is the name of this validation procedure?

I have a set of classified data. In order to test the precision of several algorithms- I split the data into train and test sets. For the test set I choose at random 30% of the data and the rest is ...
10
votes
3answers
1k views

Why isn't the holdout method (splitting data into training and testing) used in classical statistics?

In my classroom exposure to data mining, the holdout method was introduced as a way of assessing model performance. However, when I took my first class on linear models, this was not introduced as a ...
0
votes
0answers
209 views

Implementing Hopkins and Cox-Lewis index in R

I was trying to implement some clustering tendency tools in R, namely the Hopkin's index and the Cox–Lewis index. Here is the link at page 901 to show what they are This is what I managed to come up ...
5
votes
2answers
248 views

How to validate if a sample is independent and identically distributed

How can I check if the data is drawn i.i.d. from an unknown multivariate distribution?
2
votes
1answer
2k views

Classification score for Random Forest

I'm learning about the Decision Tree and Random Forests. But there is something I don't really understand. I have a training set and a cross-validation set. I need to train different Random Forests, ...
1
vote
0answers
510 views

Training/ Test Data with Time Series Model — Forecast with Training Model, or with Model based on Full Data?

Okay, I have a couple books on time series forecasting, but perhaps I need to read a couple more. Here's my question. You want to be able to validate a forecasting model. So you split the data into ...
0
votes
1answer
896 views

R times series — correct use of forecast() and accuracy() in forecast package

Cross-posting this from Stack Overflow, because it's a bit of a stats/ technology cross-over. I'm relatively new to R and the forecast package I believe authored by Rob Hyndman. I'm having trouble ...
4
votes
0answers
933 views

Hold out sample vs. cross validation for time series, and how to perform in R

I think out-of-sample validation testing for accuracy is essential in initially judging what time-series forecasts to use. In any case, I've been doing some reading on the two most common methods, ...
1
vote
1answer
75 views

Variable selection and validation dataset

According to Hastie & Tibshirani, we shouldn't use validation datasets to do variable selection; otherwise, we will overestimate the model fit. However, it seems quite often to select variables ...
0
votes
2answers
102 views

Validating clustering results with labeled data

I am working on a clustering algorithm and would like to validate its performance against a well-known and used dataset: the KDD-CUP 99 dataset ...
0
votes
0answers
56 views

Quantifying the predictive ability of a model developed from a huge data set? (variation of bootstrapping?)

I have a statistical model with around 20 predictor variables, built on 90% of a dataset consisting of over 600k observations. The original developer held out 10% of the original dataset for the ...
0
votes
2answers
81 views

Accepted Method for Selecting a Validation Set in Python

Is there an accepted method for separating out a validation set in python? In R I would use the sample function. I have 4000 training instances as json and I want ...
0
votes
0answers
691 views

How to interpret the results of bootstrapping and Monte Carlo simulation utilised to test lasso logistic regression results?

My situation: sample size: 116 binary outcome (32 events) number predictors: 42 (both continuous and categorical) predictors did not come from the top of my head; their choice was based on the ...
1
vote
0answers
414 views

Fitting Cox Regression / Proportional Hazard Model with x time interaction term in R

I am asking this in the context of wanting to diagnose for violation of proportional hazard assumption and its correction. (Schemper 1992) On p.179 of Hosmer, Lemeshow and May, it says that we can ...
0
votes
1answer
309 views

Ignore strata in external validation of stratified Cox prop hazards model?

I've fit a stratified Cox proportional hazards model to some survival data, where I've stratified by a potential confounder which is the batch the data comes from (there are three batches). Now, I'd ...
3
votes
1answer
234 views

What does a negative Somers' D say about model discriminative power?

I developed a Cox proportional hazard regression model in R. Then I tried using validate in Professor Harrell's rms package to ...
1
vote
1answer
65 views

10-fold cross-validation (high variation)

I am using 10-fold validation method to validate my model. I am using CART model and my sample size $\approx$ 50. Features $\approx$ 9. The 10-fold validated accuracy (averages) is about 76%. However, ...
1
vote
3answers
648 views

Performance drop between training and validation datasets

I have been using R's GBM (Gradient Boosting Machine) package for several months. I typically split my data into three partitions: Training, Validation, and Testing. I use the validation data set to ...
1
vote
1answer
52 views

How to validate a lognormal random walk for time series data

I am currently working on a project where I need to simulate the prices of a set of $D$ substitutable commodities over time. I was hoping to do this using the following $D$-dimensional lognormal ...
1
vote
1answer
2k views

How to use k-fold cross validation in naive bayes classifier?

I'm trying to classify text using naive bayes classifier, and also want to use k-fold cross validation to validate the result of classification. But I'm still confused how to use the k-fold cross ...
0
votes
1answer
17 views

what “non-continuous nature of scores” means

1) Can someone explain me the idea of this sentence: "It is noted that nonparametric tests were employed because of the non-continuous nature of the scale scores" (this is related to scale validity ...
0
votes
1answer
184 views

Validate a Markov Chain with few states (model switching)

Suppose I have a five state Markov chain. The states are observable (in fact I defined them, they are outcomes of an classification algorithm). So I have a long time series, see the first picture. ...
0
votes
1answer
60 views

How can one evaluate Incremental Clustering Algorithms, in particular the goodness of the clusters formed?

I have been studying an incremental clustering algorithm for a large set of data that exhibit an inherent dynamic behavior (that is new data can get added over time and some older data may get deleted ...
0
votes
0answers
12 views

What is the minimal amount of sample runs required for doing a minimum detection limit?

I have to set up an example of how to do an MDL, and the professor wants to know how many times I'd run the samples. My previous class used as little as 8, but I'm not sure if thats correct or if I ...
0
votes
0answers
348 views

All neural network designs stop because of early stopping in MATLAB

I'm using patternnet for my binary classification in MATLAB and using ...
0
votes
1answer
223 views

Validating residual plot count data (different levels)

I am studying the distribution of a marine species using the number of sightings as a dependent variable. When I am trying to validate the plots of the best model I am getting a non-usual pattern, and ...
2
votes
1answer
409 views

Out-of-sample vs. test set

Someone asked me if I did out-of-time testing (which I assume is just out-of-sample testing but with a timeline element). But if I have a test set, is that not essentially the same as out-of-sample ...
2
votes
0answers
61 views

How to deal with floor effect

I am in the process of validating a five-items scale for measuring dependence on substance 'X'. I have collected data from 98 people who used the substance under consideration at least once weekly for ...
0
votes
1answer
128 views

How do you measure the accuracy of an inference hypothesis/procedure?

Take inference to mean reasoning/predicting the value of a hidden/laten variable $Z$ given some evidence/data $X$. For example, maybe you are trying to find out if your patient has Cancer (Z = 1 if he ...
1
vote
1answer
328 views

How do you validate your machine learning models?

I am wondering what approaches are commonly used for validating machine learning models designed for classification or prediction tasks: Approaches that am using at the moment: Using truth-sets: - ...