Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

1
vote
0answers
7 views

The effect of oversampling on the positive predictive value

I need to calculate the positive predictive value for a validation set for a rare event. The problem is that the validation set was oversampled for the rare event. The event occurs in 5 percent of the ...
0
votes
0answers
28 views

External validation of a regression in Stata

What I'm essentially trying to do is a temporal external validation of a Cox Proportional Hazards model and also a logistic regression model on the newest year of a dataset that was not included ...
1
vote
0answers
33 views

k-means + linear regression: How to split the data for validation

I want to cluster my data first using k-means and then determine a regression model for each cluster. Then I want to evaluate the performance of this approach using split validation. I can think of ...
0
votes
0answers
12 views

How do I get the support vectors data (the model file) when using cross validation in LibSVM? [migrated]

I have a problem with the output file when I use ./svm-train ... -v k When I use the parameter -v, the output file doesn't get created, but I need the support ...
1
vote
0answers
28 views

Post-PCA analysis phase

Using PCA analysis, I was able to reduce the initial 23 variables into 10 principal components. But I do not understand what to do with this insight. I mean, how do I validate this information on, ...
1
vote
1answer
45 views

Binomial GLMM: Model validation & ceiling effect

My data has a binary response acc(correct/incorrect), one continuous predictor score, three categorical predictors (...
1
vote
2answers
28 views

Survey data validation with inverted questions

I have a survey using a Likert scale and two inverted questions out of twenty. Using R how can I identify (and maybe filter out) the respondents that always tick agree, for example, and thus did not ...
1
vote
0answers
17 views

rms validate on models with a predict function such as coxph and glmnet

I would like to use bootstrapping to evaluate models generated by coxph and glmnet. Would that be somehow possible with rms validate ? From the documentation it seems limited to rms functions (cph, ...
0
votes
0answers
13 views

What can be used instead of confirmatory factor analysis?

Can you please share your thoughts of the best method to deal with the following issue. In my study I'm using a well-known tool (questionnaire) which was validated in several settings but developed in ...
4
votes
2answers
84 views

How to do external validation of logistic regression models and perform model benchmarking

Quality assessment in trauma has for > 25 years been done with the US derived logistic regression model, the TRISS model. DV: survival/death and IVs: physiologic derangement (continuous), anatomic ...
2
votes
0answers
20 views

Looking for a Good Text on Statistical Analysis of Satisfaction Data

I am looking for a good textbook (or other resource) that covers the analysis of satisfaction data. Most of my data uses likert-type scales. Can anyone recommend something with examples?
0
votes
1answer
17 views

highly sporadic validation error during training with multilayer perceptron

I'm encountering an issue where a classifier I'm developing reports validation errors during training that span a wide range of values without consistently decreasing over time. Unfortunately, I'm new ...
1
vote
1answer
22 views

Using MANOVA for classification without separating training and test sets

In this study: Rosenblum, Sara, et al. "Handwriting as an objective tool for Parkinson’s disease diagnosis." Journal of neurology 260.9 (2013): 2357-2361 The researchers attempt to classify ...
1
vote
1answer
55 views

Validating the CART model in R

I have built the CART model, however I want to understand how we predict/validate the results with Validation data. ...
0
votes
0answers
27 views

Optimism bias - estimates of prediction error

The book Elements of Statistical Learning (available in PDF online) discusses the optimisim bias (7.21, page 229). It states that the optimism bias is the difference between the training error and the ...
1
vote
2answers
49 views

Model instability in data mining. When it is big enough to discredit a model and how to measure it?

Let's say I have two models. One has cumulative lift on test data 4.322578, second 2.84488. The only advantage of the second over the first consists in the quality of having the cumulative lift curve ...
0
votes
1answer
14 views

Prediction using multiple training sets

I have multiple different training sets($TS_1$,$TS_2$,..,$TS_n$) and one test set $TS$. I have calculated the prediction measures precision, recall, and F-measure for each pair ($TS_n$,$TS$). Is ...
3
votes
2answers
112 views

k folds cross validation on a multi-class dataset

Cross validation is one of the most important tools because it gives us an honest assessment of the true accuracy of our system. In other words, the cross-validation process provides a much more ...
0
votes
1answer
44 views

Repeatedly measuring accuracy against the hold out set

I have an iterative document classification task, corpus size = 300,000 documents. The labels are binary valued (yes/no). I wanted to know whether the following methodology is valid. The assumption is ...
0
votes
1answer
131 views

Accuracy rate in naive Bayes classification

I am trying to use a naive Bayes classification technique to predict fraudsters (Caller). My training set of 138 instances has 5 columns viz. ...
3
votes
0answers
45 views

Validating statistical tests for value at risk and expected shortfall

I am trying to figure out if value-at-risk (VaR, a quantile) type tests could capture if expected shortfall (expectations above a quantile) point forecast generated from a type of model could be ...
0
votes
1answer
126 views

How to validate a Multinomial Logit and Probit Model fit?

I would like to know how do you determine the performance of your models. That is, if you fit a multinomial logit or probit model for un-ordered discrete choice. What do you use to evaluate whether ...
1
vote
1answer
60 views

How to validate “equal slopes” (proportional odds) in ordinal regression

I would like to fit an ordinal regression model using proportional odds. I learned to test for "equal slopes" in order to say something about the model's validity. Therefore, I fit a model with equal ...
2
votes
1answer
51 views

Validation of a questionnaire in a new population

I have 400 responses to a 20 item questionnaire which purports to measure an attitudinal constuct in medical students. The instrument was validated in the US for a single year of medical students and ...
0
votes
0answers
35 views

holdback validation - test and validation equal but training much better - acceptable?

I have a dataset that, after modelling (with bootstrap aggregation of trees), I find I have a very high and thus over-fitted training result, but my held-back validation and test portions are ...
1
vote
0answers
20 views

Collectively evaluate a number of normal distributions [duplicate]

I build a few models, each model will produce a normal distribution for the value of a future event. For example, model M1 will produce a normal distribution $n(30, 5^2)$, and the value of the future ...
0
votes
0answers
31 views

Evaluating Expected Shortfall

I am writing my thesis on VaR and ES risk measurements and have encountered some issues with how to best test the accuracy of ES statistics. My understanding of the topic is that backtesting ES ...
3
votes
1answer
64 views

External model validation using new data for prediction: How large of a drop in $R^2$ is significant?

I need to validate a model using `external model validation' and I have a question relating to deciding when a drop in $R^2$ when compared to $R_{prediction}^2$ is significant. DISCLOSURE: This is ...
3
votes
1answer
163 views

What is the intuition behind the variation of information (VI) metric against others for cluster validation?

For non-statisticians like me, it is very difficult to capture the idea of VI metric (variation of information) even after reading the relevant paper by Marina ...
3
votes
0answers
73 views

Validation of mixed-effect models

I want to use linear mixed effect model for a set of data. After using lme4 package and lmer() function and fitting model, I want to validate my model for other ...
1
vote
0answers
45 views

How to check if the results of two methodologies are the same?

I have two models that essentially are supposed to produce the same results. Given that the first model is correct, what tests are available that I can use to check the validity of the second model ...
2
votes
0answers
63 views

Model validation in Bayesian statistics from a model with latent variables

I am working with some two-regime autoregressive models first introduced by Hamilton in 1989. The specific models is of no great concern to my question, but some variables within my autoregressive ...
0
votes
0answers
33 views

Validation of model with relatively sparse data

I am not a statistician, so please forgive my lack of precision with terminology in the following background and question. I have done a fair amount of reading and research about the following, but ...
0
votes
0answers
128 views

Cross validation for lasso logistic regression

I am writing a routine for logistic regression with lasso in matlab. So the problem is to minimize the negative log-likelihood function with the penalty term: ...
2
votes
0answers
128 views

What is the criterion of model validation?

Suppose I have a model that boasts given 1 year's daily data to calibrate the parameters, it could predict the behavior of future 1 month. What should be the criterion for back-testing or, model ...
2
votes
0answers
52 views

Ways to compare ordered binary datasets?

This question regards the best way to compare ordered binary data. My situation is as follows: I'm interested in evaluating how well a model conforms to human performance data on a set of validation ...
2
votes
2answers
118 views

Sample size for segmentation validation

I've created a automatic algorithm liver segmentation and the next step is the method validation. I have 16 exams with a mean of 40 slices each and my doubt now is how many of these slices would have ...
3
votes
2answers
152 views

How to do external validation of regression models

Very basic question here, so bear with me... I have a data set with 241 patients with 16 variables plus diagnosis (malignant vs benign). There are 3 previously published logistic regression formulas ...
1
vote
0answers
54 views

Can holdout validation be systematically biased?

I recently did some experimenting comparing some common method of internal validation. In my field, the use of a single 1:1 holdout validation is extremely common, even with very small datasets, and I ...
1
vote
1answer
192 views

GLM model validation; Patterns in residuals ? - R

I have attached an image obtained during my model validation. I fitted a negative binomial GLM to my data and I have 4 models possible with more or less the same residual plots. I have been reading ...
1
vote
1answer
138 views

How can I validate a logistic regression model using averaged parameter estimates?

Let me say thanks in advance. I'm working with a set of data that contains reported coyote sightings. I use 2/3 of the data for model calibration along with an equal number of pseudo absences. I ...
1
vote
0answers
176 views

Kendall's tau for holdouts low and not significant - Conjoint

I have done my Conjoint Analysis (fractional factorial design) but when it comes to validating the model, it shows a Kendall's tau for Holdouts of 0.33 and not significant. But Pearson's R and ...
3
votes
1answer
124 views

Statistical measures for data validation

I have to validate the data in one database by comparing it to data from a validated application system. The solution I came up with was (at least in theory) creating a statistical method that would ...
4
votes
1answer
366 views

Validate cluster analysis in R

I am trying to validate hierarchical cluster analysis result following a paper by Guy Brock, et al. clValid: An R Package for Cluster Validation (pdf). Do I have to use all these methods? What are the ...
0
votes
1answer
181 views

.632+ bootstrap estimator-defining gamma in continuous variable case

I am trying to implement the .632+ bootstrap estimator for internal validation as proposed by Efron and Tibshiraini 1997. Looking at the paper, I can see how gamma is defined in the case of ...
3
votes
1answer
225 views

Determining values of correction factor based on x bins in observed vs. actual data

I am trying to automate a problem I usually solve by hand. I have a sensor that collects data from the field. Every 6 months or so, I have to do a calibration on that sensor by collecting ...
11
votes
1answer
95 views

When building a regression model using separate modeling/validation sets, is it appropriate to “recirculate” the validation data?

Suppose I've got an 80/20 split between modeling/validation observations. I've fit a model to the modeling data set, and I'm comfortable with the error that I'm seeing on the validation data set. ...
2
votes
1answer
580 views

How do I validate my multiple linear regression model?

I have split my data into two parts. I have used my 80% data to build a multiple regression linear model. Now I want to test it using my rest 20% data. What tools on Minitab do I have to make this ...
0
votes
0answers
136 views

Mean squared error — internal and external validation

This is probably a very easy question... but I am not quite sure what the right answer should be.. I have validated 3 beforehand selected models. I have done internal cross validation and I have ...
0
votes
0answers
164 views

Validation of conditional logistic regression model with CoxReg (SPSS): must use residuals from CoxReg?

I have developed a case-control study, matched 1:2, and I have addressed the conditional logistic regression analysis with COXREG (...