Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

0
votes
0answers
18 views

How to account for drift in system identification output?

I am trying to identify the lateral dynamics of a vehicle. I use steering angle as input, and lateral displacement, lateral velocity and yaw rate as output. I use idss method of MATLAB. The issue ...
1
vote
1answer
56 views

What are the assumptions to test on unseen data to use a predictive model?

I am trying to understand what are the assumptions I have to test, to use a predictive model on unseen data. After the validation phase of my predictive model and the estimation of its accuracy on a ...
0
votes
0answers
9 views

Correcting for multiple data points

I am validating a new measure of therapist competency in psychology. I am getting 4 clients who have taken part in 40 sessions (10 each) to answer a questionnaires about their therapy sessions (...
0
votes
1answer
93 views

Calibration and validation of an accelerated failure time model on new data

I have an existing AFT (accelerated failure time) model which I'm using on a new dataset, with the obvious intent of testing whether the model predicts the new data well. A first step is to look at ...
0
votes
0answers
23 views

validation set with ranking variables

i'm working on an approach of feature selection with SVM model and i have some questions about validation , training and test sets. the idea is to rank variables in decreasing order of relevance ...
0
votes
0answers
20 views

Why use clustering + external measure for validation, instead of supervised learning?

Lets say I have set of genes in individuals and I will use clustering to find groups of individuals with similar genes and then I will check how these groups express itself in form of some disease. I ...
0
votes
0answers
20 views

How do I conduct model validation with glm models that have proportional response variable?

I would like to determine which environmental factors influence turtle nest survival. I ranked glm models with AIC using a step-wise procedure. The response variable is the proportion of eggs that ...
0
votes
0answers
25 views

Alternative to weighted kappa?

I want to compare modeled species occurrence with observed occurrence data. Using confusion matrix and weighed kappa seems to be a good option. In contrast to the normal kappa the weighted kappa takes ...
2
votes
2answers
208 views

Performing k-means clustering on a set of lines

I have a set of lines (y = numbers between 1 and 100, x= discrete) that I am trying to cluster to group similarly-shaped profiles. I have found that the profiles seem to cluster the cleanest when ...
4
votes
2answers
509 views

What is a good way to test a simple Recurrent Neural Network

I have coded up a simple real-value regression RNN in theano. What kind of dataset should I test it on? How should I go about testing it? My structure is: Univariate (for now) timeseries, $x_{...
1
vote
2answers
250 views

Is validation set always necessary?

Lets say I did the following steps: Used some separate development set to select some features. Decided a priori to use only one learning algorithm (SVM) with only default parameter values. Trained ...
2
votes
0answers
92 views

Using simulated data to check when patterns in GLMM residual plots are acceptable

I have run the following Poisson GLMM: ...
2
votes
1answer
889 views

How do you use test data set after Cross-validation

In some lectures and tutorials they suggest to split your data in three parts: training, validation and test. But it is not clear how test data set should be used and how this approach is better than ...
0
votes
0answers
45 views

Clustering technique and validation for distance based on file compression

I have a distance matrix based on a normalized compression distance between files: $$ d(x, y) = \frac{ C(xy) - \min \{ C(x), C(y) \} } {\max \{ C(x), C(y) \}} $$ Here, $C(xy)$ is the concatenation ...
7
votes
2answers
209 views

How to make sure that a machine learning algorithm's implementation is correct?

Say there is a machine learning algorithm (e.g. classification) that is well known and implemented by the original creators of the algorithm. Yet all you have is the ability to use the algorithm but ...
2
votes
1answer
65 views

Why do clustering internal validation indices are decreasing with the number of clusters?

I get the same pattern for 3 different indices: Silhouettes, Dunn and Connectivity- as the number of clusters increases, the score decreases. I am using several clustering methods and several distance ...
1
vote
1answer
40 views

Missing variable in the external validation cohort

I have developed a model on my training cohort and I intent to use an external dataset to validate the model. However, one of the predictors used in the model was not captured in the external ...
0
votes
0answers
15 views

How do I compare a new method of statistical sampling with an established one?

Lets say that: There exist 10 countries. I travel to each country and take a random subset of the population of that country and count how many of my subset are woman. I make a table of country to ...
2
votes
2answers
453 views

GLMM validation: weird qq & fit vs residual plots

I'm encountering problems with the results of a glmer model (lme4-package). Im trying to answer the question, whether a beaver ...
1
vote
1answer
80 views

K-fold cross validation for hierarchical data sets

I'm currently working on a data set that contains a hierarchical data structure (i.e., GPS locations nested within individual animals). Does anyone know how to write R code for this type of ...
1
vote
0answers
143 views

Sample size needed to validate classification/prediction model

Dose any rule of thumb exist (or possible calculation) regarding sample size needed to validate an binary classification model. We have developed this prediction model for a medical condition and ...
0
votes
1answer
51 views

Model validation using features (R)

I'm trying to create a classification model in R, at the moment i'm using 10 fold CV (using the train function of caret package) ...
0
votes
0answers
93 views

Testing model fit and running model validation plots for averaged models

I have carried out a model selection process and averaged the models within the top 2 AICcs (using MuMIn in R). I would like to test the model fit using both the chisq and validation/residual plots. ...
6
votes
1answer
1k views

Within-group sum of squares of cluster

I have a multivariate dataset for which I have only a table including the cross-wise Euclidean distances between all points and a list giving the assignment of each point to one of several clusters. ...
2
votes
1answer
204 views

Can I apply factor analysis on multiple choice questions?

I am looking to validate a questionnaire and would like to know if I can use factor analysis on the multiple-choice questions (MCQ). Also, I have another section where I am asking about participants ...
1
vote
1answer
161 views

Creating a disease severity score/index

I am trying to design a numerical scale which would describe the severity of a certain disease (in this particular case anaphylaxis). I have a set of clinical symptoms and a database of patients who ...
0
votes
0answers
33 views

multiple linear model validation

Hello I am a beginner in R. I have a dataset of fish and environmental parameters and anthropogenic pressures from 11 lakes. I have modeled some fish parameters (like fish biomass) as function of ...
3
votes
1answer
94 views

Does the position at which maximum distance occurs in a KS test make a difference?

From my understanding of the KS test, fromt the CDF of two datasets, it measures the distance between the two distributions at various points and and compares the 'maximum distance' to a predefined ...
1
vote
1answer
2k views

What is out of time validation in logistic regression model?

I understand out of sample validation very well. Can you explain what is out of time validation? Context A team in my organization has build a churn model for a teleco. Churn rate is 27%. The models ...
9
votes
4answers
1k views

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
1
vote
1answer
68 views

What is the name of this validation procedure?

I have a set of classified data. In order to test the precision of several algorithms- I split the data into train and test sets. For the test set I choose at random 30% of the data and the rest is ...
11
votes
3answers
1k views

Why isn't the holdout method (splitting data into training and testing) used in classical statistics?

In my classroom exposure to data mining, the holdout method was introduced as a way of assessing model performance. However, when I took my first class on linear models, this was not introduced as a ...
0
votes
0answers
280 views

Implementing Hopkins and Cox-Lewis index in R

I was trying to implement some clustering tendency tools in R, namely the Hopkin's index and the Cox–Lewis index. Here is the link at page 901 to show what they are This is what I managed to come up ...
5
votes
2answers
367 views

How to validate if a sample is independent and identically distributed

How can I check if the data is drawn i.i.d. from an unknown multivariate distribution?
3
votes
1answer
3k views

Classification score for Random Forest

I'm learning about the Decision Tree and Random Forests. But there is something I don't really understand. I have a training set and a cross-validation set. I need to train different Random Forests, ...
1
vote
0answers
771 views

Training/ Test Data with Time Series Model — Forecast with Training Model, or with Model based on Full Data?

Okay, I have a couple books on time series forecasting, but perhaps I need to read a couple more. Here's my question. You want to be able to validate a forecasting model. So you split the data into "...
0
votes
1answer
1k views

R times series — correct use of forecast() and accuracy() in forecast package

Cross-posting this from Stack Overflow, because it's a bit of a stats/ technology cross-over. I'm relatively new to R and the forecast package I believe authored by Rob Hyndman. I'm having trouble ...
4
votes
0answers
1k views

Hold out sample vs. cross validation for time series, and how to perform in R

I think out-of-sample validation testing for accuracy is essential in initially judging what time-series forecasts to use. In any case, I've been doing some reading on the two most common methods, ...
1
vote
1answer
86 views

Variable selection and validation dataset

According to Hastie & Tibshirani, we shouldn't use validation datasets to do variable selection; otherwise, we will overestimate the model fit. However, it seems quite often to select variables ...
0
votes
2answers
132 views

Validating clustering results with labeled data

I am working on a clustering algorithm and would like to validate its performance against a well-known and used dataset: the KDD-CUP 99 dataset (http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html)...
0
votes
0answers
59 views

Quantifying the predictive ability of a model developed from a huge data set? (variation of bootstrapping?)

I have a statistical model with around 20 predictor variables, built on 90% of a dataset consisting of over 600k observations. The original developer held out 10% of the original dataset for the ...
0
votes
2answers
122 views

Accepted Method for Selecting a Validation Set in Python

Is there an accepted method for separating out a validation set in python? In R I would use the sample function. I have 4000 training instances as json and I want ...
0
votes
0answers
836 views

How to interpret the results of bootstrapping and Monte Carlo simulation utilised to test lasso logistic regression results?

My situation: sample size: 116 binary outcome (32 events) number predictors: 42 (both continuous and categorical) predictors did not come from the top of my head; their choice was based on the ...
1
vote
0answers
502 views

Fitting Cox Regression / Proportional Hazard Model with x time interaction term in R

I am asking this in the context of wanting to diagnose for violation of proportional hazard assumption and its correction. (Schemper 1992) On p.179 of Hosmer, Lemeshow and May, it says that we can ...
0
votes
1answer
355 views

Ignore strata in external validation of stratified Cox prop hazards model?

I've fit a stratified Cox proportional hazards model to some survival data, where I've stratified by a potential confounder which is the batch the data comes from (there are three batches). Now, I'd ...
3
votes
1answer
363 views

What does a negative Somers' D say about model discriminative power?

I developed a Cox proportional hazard regression model in R. Then I tried using validate in Professor Harrell's rms package to ...
1
vote
1answer
73 views

10-fold cross-validation (high variation)

I am using 10-fold validation method to validate my model. I am using CART model and my sample size $\approx$ 50. Features $\approx$ 9. The 10-fold validated accuracy (averages) is about 76%. However, ...
1
vote
3answers
926 views

Performance drop between training and validation datasets

I have been using R's GBM (Gradient Boosting Machine) package for several months. I typically split my data into three partitions: Training, Validation, and Testing. I use the validation data set to ...
1
vote
1answer
57 views

How to validate a lognormal random walk for time series data

I am currently working on a project where I need to simulate the prices of a set of $D$ substitutable commodities over time. I was hoping to do this using the following $D$-dimensional lognormal ...
1
vote
1answer
3k views

How to use k-fold cross validation in naive bayes classifier?

I'm trying to classify text using naive bayes classifier, and also want to use k-fold cross validation to validate the result of classification. But I'm still confused how to use the k-fold cross ...