Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

4
votes
1answer
370 views

Validate cluster analysis in R

I am trying to validate hierarchical cluster analysis result following a paper by Guy Brock, et al. clValid: An R Package for Cluster Validation (pdf). Do I have to use all these methods? What are the ...
3
votes
1answer
166 views

What is the intuition behind the variation of information (VI) metric against others for cluster validation?

For non-statisticians like me, it is very difficult to capture the idea of VI metric (variation of information) even after reading the relevant paper by Marina ...
2
votes
1answer
51 views

Validation of a questionnaire in a new population

I have 400 responses to a 20 item questionnaire which purports to measure an attitudinal constuct in medical students. The instrument was validated in the US for a single year of medical students and ...
2
votes
1answer
176 views

Prediction evaluation metric for panel/longitudinal data

I would like to evaluate several different models that provide predictions of behavior at a monthly level. The data is balanced, and $n=$100,000 and $T=$12. The outcome is attending a concert in a ...
1
vote
1answer
142 views

Accuracy rate in naive Bayes classification

I am trying to use a naive Bayes classification technique to predict fraudsters (Caller). My training set of 138 instances has 5 columns viz. ...
0
votes
1answer
17 views

highly sporadic validation error during training with multilayer perceptron

I'm encountering an issue where a classifier I'm developing reports validation errors during training that span a wide range of values without consistently decreasing over time. Unfortunately, I'm new ...
0
votes
1answer
14 views

Prediction using multiple training sets

I have multiple different training sets($TS_1$,$TS_2$,..,$TS_n$) and one test set $TS$. I have calculated the prediction measures precision, recall, and F-measure for each pair ($TS_n$,$TS$). Is ...
0
votes
1answer
44 views

Repeatedly measuring accuracy against the hold out set

I have an iterative document classification task, corpus size = 300,000 documents. The labels are binary valued (yes/no). I wanted to know whether the following methodology is valid. The assumption is ...
0
votes
1answer
130 views

How to validate a Multinomial Logit and Probit Model fit?

I would like to know how do you determine the performance of your models. That is, if you fit a multinomial logit or probit model for un-ordered discrete choice. What do you use to evaluate whether ...
-1
votes
1answer
75 views

validating a regression model

I want to build a regression model to predict daily income from customers. I have 2 problems: 1. Choosing data for the training set - do I use daily income from 1 month ago, 6 month ago etc. 2. How do ...
3
votes
0answers
45 views

Validating statistical tests for value at risk and expected shortfall

I am trying to figure out if value-at-risk (VaR, a quantile) type tests could capture if expected shortfall (expectations above a quantile) point forecast generated from a type of model could be ...
3
votes
0answers
73 views

Validation of mixed-effect models

I want to use linear mixed effect model for a set of data. After using lme4 package and lmer() function and fitting model, I want to validate my model for other ...
3
votes
0answers
266 views

Logistic Regression Cost Function issue in Matlab

I'm trying to implement a logistic regression function in matlab. I calculated the theta values, linear regression cost function is converging and then I use those parameters in logistic regression ...
3
votes
0answers
94 views

Validating a logistic regression for a specific $x$

I have a logistic regression model for 0/1 binary response data that is built from samples $(x_1,Y_1),\ldots,(x_m,Y_m)$, where $x_1,\ldots,x_m$ are, fixed, nonrandom, real values and $Y_1,\ldots,Y_m$ ...
2
votes
0answers
20 views

Looking for a Good Text on Statistical Analysis of Satisfaction Data

I am looking for a good textbook (or other resource) that covers the analysis of satisfaction data. Most of my data uses likert-type scales. Can anyone recommend something with examples?
2
votes
0answers
64 views

Model validation in Bayesian statistics from a model with latent variables

I am working with some two-regime autoregressive models first introduced by Hamilton in 1989. The specific models is of no great concern to my question, but some variables within my autoregressive ...
2
votes
0answers
128 views

What is the criterion of model validation?

Suppose I have a model that boasts given 1 year's daily data to calibrate the parameters, it could predict the behavior of future 1 month. What should be the criterion for back-testing or, model ...
2
votes
0answers
52 views

Ways to compare ordered binary datasets?

This question regards the best way to compare ordered binary data. My situation is as follows: I'm interested in evaluating how well a model conforms to human performance data on a set of validation ...
2
votes
0answers
72 views

What tasks should I do when performing a model validation?

I have a mixed model built using Data A below, and now I want to validate the model using Data B. What should I do to actually "validate" the model? ...
2
votes
0answers
89 views

Validation of flow cytometer (Facs Canto II)

I was just wondering if 3 biological samples would be statiscally significant for both precision testing and calibration verification. Precision: to check whether instrument is able to give ...
2
votes
0answers
49 views

Validating previous analysis with new data

I am interested in trying to validate some results from a previous experiment. The experiment involves selecting probes of interest that separate our two populations of interest, e.g. smoker vs. ...
2
votes
0answers
93 views

Validation of data partitioning

I've some numerical sequence of data monitored over time $v(i)$, which belongs to one of two classes, 0 or 1. I monitor the variable several times, obtaining a sequence associated to 0 or 1. I use ...
1
vote
0answers
11 views

The effect of oversampling on the positive predictive value

I need to calculate the positive predictive value for a validation set for a rare event. The problem is that the validation set was oversampled for the rare event. The event occurs in 5 percent of the ...
1
vote
0answers
33 views

k-means + linear regression: How to split the data for validation

I want to cluster my data first using k-means and then determine a regression model for each cluster. Then I want to evaluate the performance of this approach using split validation. I can think of ...
1
vote
0answers
29 views

Post-PCA analysis phase

Using PCA analysis, I was able to reduce the initial 23 variables into 10 principal components. But I do not understand what to do with this insight. I mean, how do I validate this information on, ...
1
vote
0answers
19 views

rms validate on models with a predict function such as coxph and glmnet

I would like to use bootstrapping to evaluate models generated by coxph and glmnet. Would that be somehow possible with rms validate ? From the documentation it seems limited to rms functions (cph, ...
1
vote
0answers
45 views

How to check if the results of two methodologies are the same?

I have two models that essentially are supposed to produce the same results. Given that the first model is correct, what tests are available that I can use to check the validity of the second model ...
1
vote
0answers
55 views

Can holdout validation be systematically biased?

I recently did some experimenting comparing some common method of internal validation. In my field, the use of a single 1:1 holdout validation is extremely common, even with very small datasets, and I ...
1
vote
0answers
177 views

Kendall's tau for holdouts low and not significant - Conjoint

I have done my Conjoint Analysis (fractional factorial design) but when it comes to validating the model, it shows a Kendall's tau for Holdouts of 0.33 and not significant. But Pearson's R and ...
1
vote
0answers
64 views

Assessment of a single predictive model

I am facing a situation where I need to assess the predictive ability of a single (regression) model. I do not need to compare its performance with some other model, I just need to show that it ...
1
vote
0answers
423 views

Validation error less than training error — implications?

I am running a neural net to predict used car prices, sample size is 800. Using both 10-fold cross validation (10 times) and 1/3 holdback (10 times), the $R^2$ for training is about 0.60 and for ...
1
vote
0answers
99 views

Intraclass correlation to validate a measure

I would like to understand whether the output of an algorithm I developed agrees with the ratings of external observers. The number of observers is 10 and each of them rated on 2 3-point-Likert items ...
1
vote
0answers
158 views

Minimal number of samples/conversions for statistical validity

We are measuring conversion rates (% of visitors who bought) on an e-commerce site. The test apply to a segment of visitors who meet specific criteria (for example people from a certain country). ...
0
votes
0answers
7 views

How to validate classifier (built by using MLN method)?

I have developed a method (let's call it Method X) that has a classifier function. The classifier function was built by using MLN (Markov logic network). I need to ...
0
votes
0answers
31 views

External validation of a regression in Stata

What I'm essentially trying to do is a temporal external validation of a Cox Proportional Hazards model and also a logistic regression model on the newest year of a dataset that was not included ...
0
votes
0answers
13 views

What can be used instead of confirmatory factor analysis?

Can you please share your thoughts of the best method to deal with the following issue. In my study I'm using a well-known tool (questionnaire) which was validated in several settings but developed in ...
0
votes
0answers
28 views

Optimism bias - estimates of prediction error

The book Elements of Statistical Learning (available in PDF online) discusses the optimisim bias (7.21, page 229). It states that the optimism bias is the difference between the training error and the ...
0
votes
0answers
35 views

holdback validation - test and validation equal but training much better - acceptable?

I have a dataset that, after modelling (with bootstrap aggregation of trees), I find I have a very high and thus over-fitted training result, but my held-back validation and test portions are ...
0
votes
0answers
31 views

Evaluating Expected Shortfall

I am writing my thesis on VaR and ES risk measurements and have encountered some issues with how to best test the accuracy of ES statistics. My understanding of the topic is that backtesting ES ...
0
votes
0answers
33 views

Validation of model with relatively sparse data

I am not a statistician, so please forgive my lack of precision with terminology in the following background and question. I have done a fair amount of reading and research about the following, but ...
0
votes
0answers
130 views

Cross validation for lasso logistic regression

I am writing a routine for logistic regression with lasso in matlab. So the problem is to minimize the negative log-likelihood function with the penalty term: ...
0
votes
0answers
137 views

Mean squared error — internal and external validation

This is probably a very easy question... but I am not quite sure what the right answer should be.. I have validated 3 beforehand selected models. I have done internal cross validation and I have ...
0
votes
0answers
165 views

Validation of conditional logistic regression model with CoxReg (SPSS): must use residuals from CoxReg?

I have developed a case-control study, matched 1:2, and I have addressed the conditional logistic regression analysis with COXREG (...
0
votes
0answers
308 views

How to write a loop in R to select multiple regression model and validate it?

I would like to run a loop in R. I have never done this before, so I would be very grateful for your help ! I have a sample set: 25 objects. I would like to draw 1 object from it and use it as a ...
0
votes
0answers
362 views

Method to output statistical score when validating using bootstrap method on ordinal logistic regression model

It is about sas programming. Thank you in advance. I build an ordinal logistic regression model on my data. ...