Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

0
votes
0answers
13 views

Validate a Markov Chain with few states (model switching)

Suppose I have a five state Markov chain. The states are observable (in fact I defined them, they are outcomes of an classification algorithm). So I have a long time series, see the first picture. ...
4
votes
1answer
615 views

Validate cluster analysis in R

I am trying to validate hierarchical cluster analysis result following a paper by Guy Brock, et al. clValid: An R Package for Cluster Validation (pdf). Do I have to use all these methods? What are the ...
0
votes
1answer
18 views

How can one evaluate Incremental Clustering Algorithms, in particular the goodness of the clusters formed?

I have been studying an incremental clustering algorithm for a large set of data that exhibit an inherent dynamic behavior (that is new data can get added over time and some older data may get deleted ...
4
votes
1answer
341 views

What is the intuition behind the variation of information (VI) metric for cluster validation?

For non-statisticians like me, it is very difficult to capture the idea of VI metric (variation of information) even after reading the relevant paper by Marina ...
2
votes
1answer
220 views

Prediction evaluation metric for panel/longitudinal data

I would like to evaluate several different models that provide predictions of behavior at a monthly level. The data is balanced, and $n=$100,000 and $T=$12. The outcome is attending a concert in a ...
0
votes
0answers
9 views

What is the minimal amount of sample runs required for doing a minimum detection limit?

I have to set up an example of how to do an MDL, and the professor wants to know how many times I'd run the samples. My previous class used as little as 8, but I'm not sure if thats correct or if I ...
1
vote
1answer
120 views

How do you validate your machine learning models?

I am wondering what approaches are commonly used for validating machine learning models designed for classification or prediction tasks: Approaches that am using at the moment: Using truth-sets: - ...
0
votes
1answer
30 views

Validating residual plot count data (different levels)

I am studying the distribution of a marine species using the number of sightings as a dependent variable. When I am trying to validate the plots of the best model I am getting a non-usual pattern, and ...
3
votes
1answer
97 views

Validation of a questionnaire in a new population

I have 400 responses to a 20 item questionnaire which purports to measure an attitudinal constuct in medical students. The instrument was validated in the US for a single year of medical students and ...
0
votes
0answers
13 views
0
votes
1answer
72 views

Repeatedly measuring accuracy against the hold out set

I have an iterative document classification task, corpus size = 300,000 documents. The labels are binary valued (yes/no). I wanted to know whether the following methodology is valid. The assumption is ...
0
votes
0answers
38 views

Validating Bootstrapped Probability of Survival Results From Small Sample Size Data

Quite often in industry, due to cost and schedule constraints, decisions must be made on small sample size data. I have 4 cycles-to-failure values resulting from running samples to failure in a ...
0
votes
0answers
12 views

Model validation and verification for Markov Chain switching model

Assuming I have a discrete-time Markov chain with only five states. The chain will be used for the prediction of the macroscopic states which are observable and coming from a timeseries. I use maximum ...
2
votes
1answer
62 views

Cross-validation for Comparing Clustering Techniques

I'm working on comparing multiple clustering algorithms to each other using the adjusted Rand index for a given dataset. We have a gold standard that we'd like to compare the obtained clustering ...
5
votes
1answer
126 views

Validating a logistic regression for a specific $x$

I have a logistic regression model for 0/1 binary response data that is built from samples $(x_1,Y_1),\dots,(x_m,Y_m)$, where $x_1,\dots,x_m$ are, fixed, nonrandom, real values and $Y_1,\dots,Y_m$ are ...
0
votes
1answer
18 views

rms package, getting zero Dxy in validation of cph model

I'm doing Cox proportional hazards analysis using Frank Harrell's rms package (4.2.0), with 3 strata. My model seems reasonably predictive on training data (Dxy=-0.537), but somehow in bootstrap ...
2
votes
0answers
38 views

How to deal with floor effect

I am in the process of validating a five-items scale for measuring dependence on substance 'X'. I have collected data from 98 people who used the substance under consideration at least once weekly for ...
0
votes
0answers
47 views

How to validate goodness of fit and forecasting quality of a model?

I am working on big data sets, which represent electricity consumption on power substation throughout the year. I have data every 10 minutes and multiple (and long) seasonalities. I have a daily, ...
1
vote
1answer
27 views

Out-of-sample vs. test set

Someone asked me if I did out-of-time testing (which I assume is just out-of-sample testing but with a timeline element). But if I have a test set, is that not essentially the same as out-of-sample ...
0
votes
1answer
45 views

How do you measure the accuracy of an inference hypothesis/procedure?

Take inference to mean reasoning/predicting the value of a hidden/laten variable $Z$ given some evidence/data $X$. For example, maybe you are trying to find out if your patient has Cancer (Z = 1 if he ...
0
votes
0answers
49 views

Compare LMM GLMM (generalised linear mixed model, negative binomial) by numerical measure (AIC BIC, cross validation, R² squared) for model validation

How to compare results of generalized linear mixed model (GLMM, negative binomial) with a log transformed linear mixed model (multilevel, hierarchical) . I have a data set (counts), which is nested. ...
0
votes
0answers
13 views

Applying Cox proportional hazards model to new data to get absolute risk (Framingham risk score)

I'm trying to apply one of the Framingham cardiovascular event risk scores to a new dataset in order to get absolute risk. D'Agostino 2013 "Cardiovascular Disease Risk Assessment: Insights from ...
1
vote
0answers
12 views

Derive minimum positive rate of change for co2 data

I have CO2 (in parts per million) data of a closed room. The CO2 data is recorded along with timestamps. Typical difference between two samples is around five minutes. My aim is to find occupancy of ...
2
votes
2answers
38 views

The size of the sample for split validation

At this moment I have a dataset with 4000 samples (50% positive and 50% negative). Normally I would do cross validation for this approach, however besides normal data mining techniques I am also ...
1
vote
3answers
809 views

How to validate a Multinomial Logit and Probit Model fit?

I would like to know how do you determine the performance of your models. That is, if you fit a multinomial logit or probit model for un-ordered discrete choice. What do you use to evaluate whether ...
1
vote
2answers
165 views

Logistic Regression Model Validation

I am validating a logistic regression model. This is the first time i am validating a model. I am using split sampling method. I have split data randomly into two parts - 70% development and 30% ...
0
votes
0answers
16 views

Validating a multivariate categorical model

I assume that my population is a sample of an unknown multivariate categorical distribution $\mathbf{X} = (X_1, X_2, \ldots, X_k)$. From this population, a sample $\mathbf{X^*}$ is available, I assume ...
1
vote
1answer
833 views

Accuracy rate in naive Bayes classification

I am trying to use a naive Bayes classification technique to predict fraudsters (Caller). My training set of 138 instances has 5 columns viz. ...
0
votes
0answers
50 views

Validation - correctly compare and validated imputation models

I've seen a lot of interesting questions here about multiple imputation and also great answers that helped me a lot to impute my data. I've used Predictive Mean Matching, EMB and I would like to use ...
0
votes
1answer
24 views

Computing predicition intervals with cross-validation?

I'm using a k-fold (10-fold) cross-validation while building a model. I'm only using it to get an estimate of the out-of-sample error, not to pick a model from candidates. For example, if I have 30 ...
1
vote
1answer
51 views

Is the statistical significance of a regression meaningful if it has poor out of sample performance?

I want to determine the significance of a particular variable, among many confounders. If I fit a model on the training set and observe a small p value, should I discard the model because it ...
3
votes
2answers
1k views

How do I validate my multiple linear regression model?

I have split my data into two parts. I have used my 80% data to build a multiple regression linear model. Now I want to test it using my rest 20% data. What tools on Minitab do I have to make this ...
4
votes
0answers
126 views

Are world cup predictions testable?

As of today, dozens of soccer world cup predictions exist, some more complex, some more elegant, and most of them predict every nation's "chance" of winning a particular match/ the cup. As I am ...
3
votes
0answers
115 views

Internal validation via bootstrap: What ROC curve to present?

I am using the bootstrap approach for internal validation of a multivariate model built with either standard logistic regression OR elastic net. The procedure I use is as follows: 1) build model ...
-1
votes
1answer
14 views

On which data should the lift be calculated i.e. Training set or Test set and why?

On which data should the lift be calculated i.e. Training set or Test set and why? What does the lift value 115% mean
2
votes
2answers
62 views

Split train//validation/test sets by time, is it correct?

Here's the scenario, slightly altered to a common one. Credit card fraud, payments for the last 12 months (a rolling window). Train with the data from the first 10 months, validate with data from the ...
0
votes
1answer
30 views

use of validation set on lasso cross validation

When training a model a train, a validation and test set are used. I was wondering if there is any paper or example that proves that the use of an independent validation set increase the performance ...
0
votes
0answers
44 views

How would you validate a random walk model?

I have used a random walk model and Gibbs sampling (more specifically RJAGS) in order to obtain posterior of the state given the observations. In this case the state is the true proportion of the ...
0
votes
0answers
23 views

Incorporating validation data into training set

Suppose that I divide my data for modelling purposes into training, testing, and validation which will then be deployed for an application (as in the response to this question). Why not incorporate ...
34
votes
5answers
25k views

What is the difference between test set and validation set?

I found this confusing when I use the neural network toolbox in Matlab. It divided the raw data set into three parts: training set validation set test set I notice in many training or learning ...
0
votes
1answer
29 views

highly sporadic validation error during training with multilayer perceptron

I'm encountering an issue where a classifier I'm developing reports validation errors during training that span a wide range of values without consistently decreasing over time. Unfortunately, I'm new ...
0
votes
0answers
15 views

How to validate classifier (built by using MLN method)?

I have developed a method (let's call it Method X) that has a classifier function. The classifier function was built by using MLN (Markov logic network). I need to ...
1
vote
0answers
34 views

The effect of oversampling on the positive predictive value

I need to calculate the positive predictive value for a validation set for a rare event. The problem is that the validation set was oversampled for the rare event. The event occurs in 5 percent of the ...
0
votes
0answers
99 views

External validation of a regression in Stata

What I'm essentially trying to do is a temporal external validation of a Cox Proportional Hazards model and also a logistic regression model on the newest year of a dataset that was not included ...
1
vote
2answers
76 views

Model instability in data mining. When it is big enough to discredit a model and how to measure it?

Let's say I have two models. One has cumulative lift on test data 4.322578, second 2.84488. The only advantage of the second over the first consists in the quality of having the cumulative lift curve ...
1
vote
0answers
123 views

k-means + linear regression: How to split the data for validation

I want to cluster my data first using k-means and then determine a regression model for each cluster. Then I want to evaluate the performance of this approach using split validation. I can think of ...
0
votes
1answer
18 views

Prediction using multiple training sets

I have multiple different training sets($TS_1$,$TS_2$,..,$TS_n$) and one test set $TS$. I have calculated the prediction measures precision, recall, and F-measure for each pair ($TS_n$,$TS$). Is ...
1
vote
0answers
32 views

Post-PCA analysis phase

Using PCA analysis, I was able to reduce the initial 23 variables into 10 principal components. But I do not understand what to do with this insight. I mean, how do I validate this information on, ...
1
vote
1answer
141 views

Validating the CART model in R

I have built the CART model, however I want to understand how we predict/validate the results with Validation data. ...
1
vote
1answer
113 views

Binomial GLMM: Model validation & ceiling effect

My data has a binary response acc(correct/incorrect), one continuous predictor score, three categorical predictors (...