Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

1
vote
0answers
23 views

How to deal with floor effect

I am in the process of validating a five-items scale for measuring dependence on substance 'X'. I have collected data from 98 people who used the substance under consideration at least once weekly for ...
0
votes
1answer
29 views

How do you measure the accuracy of an inference hypothesis/procedure?

Take inference to mean reasoning/predicting the value of a hidden/laten variable $Z$ given some evidence/data $X$. For example, maybe you are trying to find out if your patient has Cancer (Z = 1 if he ...
0
votes
0answers
15 views

Compare LMM GLMM (generalised linear mixed model, negative binomial) by numerical measure (AIC BIC, cross validation, R² squared) for model validation

How to compare results of generalized linear mixed model (GLMM, negative binomial) with a log transformed linear mixed model (multilevel, hierarchical) . I have a data set (counts), which is nested. ...
0
votes
0answers
4 views

Applying Cox proportional hazards model to new data to get absolute risk (Framingham risk score)

I'm trying to apply one of the Framingham cardiovascular event risk scores to a new dataset in order to get absolute risk. D'Agostino 2013 "Cardiovascular Disease Risk Assessment: Insights from ...
1
vote
1answer
65 views

How do you validate your machine learning models?

I am wondering what approaches are commonly used for validating a classification or prediction models: Approaches that am using at the moment: Using truth-sets: - ROCs, Bootstrapping, Accuracy, ...
1
vote
0answers
10 views

Derive minimum positive rate of change for co2 data

I have CO2 (in parts per million) data of a closed room. The CO2 data is recorded along with timestamps. Typical difference between two samples is around five minutes. My aim is to find occupancy of ...
2
votes
2answers
32 views

The size of the sample for split validation

At this moment I have a dataset with 4000 samples (50% positive and 50% negative). Normally I would do cross validation for this approach, however besides normal data mining techniques I am also ...
1
vote
2answers
84 views

Logistic Regression Model Validation

I am validating a logistic regression model. This is the first time i am validating a model. I am using split sampling method. I have split data randomly into two parts - 70% development and 30% ...
0
votes
0answers
12 views

Validating a multivariate categorical model

I assume that my population is a sample of an unknown multivariate categorical distribution $\mathbf{X} = (X_1, X_2, \ldots, X_k)$. From this population, a sample $\mathbf{X^*}$ is available, I assume ...
1
vote
1answer
50 views

Is the statistical significance of a regression meaningful if it has poor out of sample performance?

I want to determine the significance of a particular variable, among many confounders. If I fit a model on the training set and observe a small p value, should I discard the model because it ...
0
votes
0answers
45 views

Validation - correctly compare and validated imputation models

I've seen a lot of interesting questions here about multiple imputation and also great answers that helped me a lot to impute my data. I've used Predictive Mean Matching, EMB and I would like to use ...
4
votes
0answers
119 views

Are world cup predictions testable?

As of today, dozens of soccer world cup predictions exist, some more complex, some more elegant, and most of them predict every nation's "chance" of winning a particular match/ the cup. As I am ...
3
votes
0answers
56 views

Internal validation via bootstrap: What ROC curve to present?

I am using the bootstrap approach for internal validation of a multivariate model built with either standard logistic regression OR elastic net. The procedure I use is as follows: 1) build model ...
0
votes
1answer
22 views

Computing predicition intervals with cross-validation?

I'm using a k-fold (10-fold) cross-validation while building a model. I'm only using it to get an estimate of the out-of-sample error, not to pick a model from candidates. For example, if I have 30 ...
2
votes
2answers
58 views

Split train//validation/test sets by time, is it correct?

Here's the scenario, slightly altered to a common one. Credit card fraud, payments for the last 12 months (a rolling window). Train with the data from the first 10 months, validate with data from the ...
-1
votes
1answer
12 views

On which data should the lift be calculated i.e. Training set or Test set and why?

On which data should the lift be calculated i.e. Training set or Test set and why? What does the lift value 115% mean
0
votes
0answers
43 views

How would you validate a random walk model?

I have used a random walk model and Gibbs sampling (more specifically RJAGS) in order to obtain posterior of the state given the observations. In this case the state is the true proportion of the ...
0
votes
1answer
26 views

use of validation set on lasso cross validation

When training a model a train, a validation and test set are used. I was wondering if there is any paper or example that proves that the use of an independent validation set increase the performance ...
0
votes
0answers
21 views

Incorporating validation data into training set

Suppose that I divide my data for modelling purposes into training, testing, and validation which will then be deployed for an application (as in the response to this question). Why not incorporate ...
2
votes
1answer
49 views

Cross-validation for Comparing Clustering Techniques

I'm working on comparing multiple clustering algorithms to each other using the adjusted Rand index for a given dataset. We have a gold standard that we'd like to compare the obtained clustering ...
0
votes
0answers
12 views

How to validate classifier (built by using MLN method)?

I have developed a method (let's call it Method X) that has a classifier function. The classifier function was built by using MLN (Markov logic network). I need to ...
1
vote
0answers
30 views

The effect of oversampling on the positive predictive value

I need to calculate the positive predictive value for a validation set for a rare event. The problem is that the validation set was oversampled for the rare event. The event occurs in 5 percent of the ...
0
votes
0answers
81 views

External validation of a regression in Stata

What I'm essentially trying to do is a temporal external validation of a Cox Proportional Hazards model and also a logistic regression model on the newest year of a dataset that was not included ...
1
vote
0answers
94 views

k-means + linear regression: How to split the data for validation

I want to cluster my data first using k-means and then determine a regression model for each cluster. Then I want to evaluate the performance of this approach using split validation. I can think of ...
1
vote
0answers
32 views

Post-PCA analysis phase

Using PCA analysis, I was able to reduce the initial 23 variables into 10 principal components. But I do not understand what to do with this insight. I mean, how do I validate this information on, ...
1
vote
1answer
88 views

Binomial GLMM: Model validation & ceiling effect

My data has a binary response acc(correct/incorrect), one continuous predictor score, three categorical predictors (...
1
vote
2answers
60 views

Survey data validation with inverted questions

I have a survey using a Likert scale and two inverted questions out of twenty. Using R how can I identify (and maybe filter out) the respondents that always tick agree, for example, and thus did not ...
1
vote
0answers
39 views

rms validate on models with a predict function such as coxph and glmnet

I would like to use bootstrapping to evaluate models generated by coxph and glmnet. Would that be somehow possible with rms validate ? From the documentation it seems limited to rms functions (cph, ...
0
votes
0answers
23 views

What can be used instead of confirmatory factor analysis?

Can you please share your thoughts of the best method to deal with the following issue. In my study I'm using a well-known tool (questionnaire) which was validated in several settings but developed in ...
5
votes
2answers
136 views

How to do external validation of logistic regression models and perform model benchmarking

Quality assessment in trauma has for > 25 years been done with the US derived logistic regression model, the TRISS model. DV: survival/death and IVs: physiologic derangement (continuous), anatomic ...
2
votes
0answers
23 views

Looking for a Good Text on Statistical Analysis of Satisfaction Data

I am looking for a good textbook (or other resource) that covers the analysis of satisfaction data. Most of my data uses likert-type scales. Can anyone recommend something with examples?
0
votes
1answer
27 views

highly sporadic validation error during training with multilayer perceptron

I'm encountering an issue where a classifier I'm developing reports validation errors during training that span a wide range of values without consistently decreasing over time. Unfortunately, I'm new ...
1
vote
1answer
33 views

Using MANOVA for classification without separating training and test sets

In this study: Rosenblum, Sara, et al. "Handwriting as an objective tool for Parkinson’s disease diagnosis." Journal of neurology 260.9 (2013): 2357-2361 The researchers attempt to classify ...
1
vote
1answer
99 views

Validating the CART model in R

I have built the CART model, however I want to understand how we predict/validate the results with Validation data. ...
0
votes
0answers
49 views

Optimism bias - estimates of prediction error

The book Elements of Statistical Learning (available in PDF online) discusses the optimisim bias (7.21, page 229). It states that the optimism bias is the difference between the training error and the ...
1
vote
2answers
69 views

Model instability in data mining. When it is big enough to discredit a model and how to measure it?

Let's say I have two models. One has cumulative lift on test data 4.322578, second 2.84488. The only advantage of the second over the first consists in the quality of having the cumulative lift curve ...
0
votes
1answer
17 views

Prediction using multiple training sets

I have multiple different training sets($TS_1$,$TS_2$,..,$TS_n$) and one test set $TS$. I have calculated the prediction measures precision, recall, and F-measure for each pair ($TS_n$,$TS$). Is ...
3
votes
2answers
235 views

k folds cross validation on a multi-class dataset

Cross validation is one of the most important tools because it gives us an honest assessment of the true accuracy of our system. In other words, the cross-validation process provides a much more ...
0
votes
1answer
58 views

Repeatedly measuring accuracy against the hold out set

I have an iterative document classification task, corpus size = 300,000 documents. The labels are binary valued (yes/no). I wanted to know whether the following methodology is valid. The assumption is ...
1
vote
1answer
599 views

Accuracy rate in naive Bayes classification

I am trying to use a naive Bayes classification technique to predict fraudsters (Caller). My training set of 138 instances has 5 columns viz. ...
3
votes
0answers
62 views

Validating statistical tests for value at risk and expected shortfall

I am trying to figure out if value-at-risk (VaR, a quantile) type tests could capture if expected shortfall (expectations above a quantile) point forecast generated from a type of model could be ...
1
vote
3answers
581 views

How to validate a Multinomial Logit and Probit Model fit?

I would like to know how do you determine the performance of your models. That is, if you fit a multinomial logit or probit model for un-ordered discrete choice. What do you use to evaluate whether ...
1
vote
1answer
74 views

How to validate “equal slopes” (proportional odds) in ordinal regression

I would like to fit an ordinal regression model using proportional odds. I learned to test for "equal slopes" in order to say something about the model's validity. Therefore, I fit a model with equal ...
3
votes
1answer
84 views

Validation of a questionnaire in a new population

I have 400 responses to a 20 item questionnaire which purports to measure an attitudinal constuct in medical students. The instrument was validated in the US for a single year of medical students and ...
0
votes
0answers
48 views

holdback validation - test and validation equal but training much better - acceptable?

I have a dataset that, after modelling (with bootstrap aggregation of trees), I find I have a very high and thus over-fitted training result, but my held-back validation and test portions are ...
1
vote
0answers
20 views

Collectively evaluate a number of normal distributions [duplicate]

I build a few models, each model will produce a normal distribution for the value of a future event. For example, model M1 will produce a normal distribution $n(30, 5^2)$, and the value of the future ...
0
votes
0answers
44 views

Evaluating Expected Shortfall

I am writing my thesis on VaR and ES risk measurements and have encountered some issues with how to best test the accuracy of ES statistics. My understanding of the topic is that backtesting ES ...
3
votes
1answer
97 views

External model validation using new data for prediction: How large of a drop in $R^2$ is significant?

I need to validate a model using `external model validation' and I have a question relating to deciding when a drop in $R^2$ when compared to $R_{prediction}^2$ is significant. DISCLOSURE: This is ...
4
votes
1answer
290 views

What is the intuition behind the variation of information (VI) metric for cluster validation?

For non-statisticians like me, it is very difficult to capture the idea of VI metric (variation of information) even after reading the relevant paper by Marina ...
3
votes
0answers
83 views

Validation of mixed-effect models

I want to use linear mixed effect model for a set of data. After using lme4 package and lmer() function and fitting model, I want to validate my model for other ...