Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

5
votes
0answers
419 views

Internal validation via bootstrap: What ROC curve to present?

I am using the bootstrap approach for internal validation of a multivariate model built with either standard logistic regression OR elastic net. The procedure I use is as follows: 1) build model ...
4
votes
0answers
143 views

Are world cup predictions testable?

As of today, dozens of soccer world cup predictions exist, some more complex, some more elegant, and most of them predict every nation's "chance" of winning a particular match/ the cup. As I am ...
3
votes
0answers
131 views

Validating statistical tests for value at risk and expected shortfall

I am trying to figure out if value-at-risk (VaR, a quantile) type tests could capture if expected shortfall (expectations above a quantile) point forecast generated from a type of model could be ...
3
votes
0answers
115 views

Validation of mixed-effect models

I want to use linear mixed effect model for a set of data. After using lme4 package and lmer() function and fitting model, I want to validate my model for other ...
3
votes
0answers
461 views

Logistic Regression Cost Function issue in Matlab

I'm trying to implement a logistic regression function in matlab. I calculated the theta values, linear regression cost function is converging and then I use those parameters in logistic regression ...
2
votes
0answers
53 views

Using simulated data to check when patterns in GLMM residual plots are acceptable

I have run the following Poisson GLMM: ...
2
votes
0answers
53 views

How to deal with floor effect

I am in the process of validating a five-items scale for measuring dependence on substance 'X'. I have collected data from 98 people who used the substance under consideration at least once weekly for ...
2
votes
0answers
31 views

Looking for a Good Text on Statistical Analysis of Satisfaction Data

I am looking for a good textbook (or other resource) that covers the analysis of satisfaction data. Most of my data uses likert-type scales. Can anyone recommend something with examples?
2
votes
0answers
115 views

Model validation in Bayesian statistics from a model with latent variables

I am working with some two-regime autoregressive models first introduced by Hamilton in 1989. The specific models is of no great concern to my question, but some variables within my autoregressive ...
2
votes
0answers
145 views

What is the criterion of model validation?

Suppose I have a model that boasts given 1 year's daily data to calibrate the parameters, it could predict the behavior of future 1 month. What should be the criterion for back-testing or, model ...
2
votes
0answers
61 views

Ways to compare ordered binary datasets?

This question regards the best way to compare ordered binary data. My situation is as follows: I'm interested in evaluating how well a model conforms to human performance data on a set of validation ...
2
votes
0answers
79 views

What tasks should I do when performing a model validation?

I have a mixed model built using Data A below, and now I want to validate the model using Data B. What should I do to actually "validate" the model? ...
2
votes
0answers
143 views

Validation of flow cytometer (Facs Canto II)

I was just wondering if 3 biological samples would be statiscally significant for both precision testing and calibration verification. Precision: to check whether instrument is able to give ...
2
votes
0answers
51 views

Validating previous analysis with new data

I am interested in trying to validate some results from a previous experiment. The experiment involves selecting probes of interest that separate our two populations of interest, e.g. smoker vs. ...
2
votes
0answers
99 views

Validation of data partitioning

I've some numerical sequence of data monitored over time $v(i)$, which belongs to one of two classes, 0 or 1. I monitor the variable several times, obtaining a sequence associated to 0 or 1. I use ...
1
vote
0answers
57 views

Sample size needed to validate classification/prediction model

Dose any rule of thumb exist (or possible calculation) regarding sample size needed to validate an binary classification model. We have developed this prediction model for a medical condition and ...
1
vote
0answers
251 views

Training/ Test Data with Time Series Model — Forecast with Training Model, or with Model based on Full Data?

Okay, I have a couple books on time series forecasting, but perhaps I need to read a couple more. Here's my question. You want to be able to validate a forecasting model. So you split the data into ...
1
vote
0answers
343 views

Fitting Cox Regression / Proportional Hazard Model with x time interaction term in R

I am asking this in the context of wanting to diagnose for violation of proportional hazard assumption and its correction. (Schemper 1992) On p.179 of Hosmer, Lemeshow and May, it says that we can ...
1
vote
0answers
20 views

Derive minimum positive rate of change for co2 data

I have CO2 (in parts per million) data of a closed room. The CO2 data is recorded along with timestamps. Typical difference between two samples is around five minutes. My aim is to find occupancy of ...
1
vote
0answers
61 views

The effect of oversampling on the positive predictive value

I need to calculate the positive predictive value for a validation set for a rare event. The problem is that the validation set was oversampled for the rare event. The event occurs in 5 percent of the ...
1
vote
0answers
360 views

k-means + linear regression: How to split the data for validation

I want to cluster my data first using k-means and then determine a regression model for each cluster. Then I want to evaluate the performance of this approach using split validation. I can think of ...
1
vote
0answers
76 views

Evaluating Expected Shortfall

I am writing my thesis on VaR and ES risk measurements and have encountered some issues with how to best test the accuracy of ES statistics. My understanding of the topic is that backtesting ES ...
1
vote
0answers
48 views

How to check if the results of two methodologies are the same?

I have two models that essentially are supposed to produce the same results. Given that the first model is correct, what tests are available that I can use to check the validity of the second model ...
1
vote
0answers
102 views

Can holdout validation be systematically biased?

I recently did some experimenting comparing some common method of internal validation. In my field, the use of a single 1:1 holdout validation is extremely common, even with very small datasets, and I ...
1
vote
0answers
461 views

Kendall's tau for holdouts low and not significant - Conjoint

I have done my Conjoint Analysis (fractional factorial design) but when it comes to validating the model, it shows a Kendall's tau for Holdouts of 0.33 and not significant. But Pearson's R and ...
1
vote
0answers
80 views

Assessment of a single predictive model

I am facing a situation where I need to assess the predictive ability of a single (regression) model. I do not need to compare its performance with some other model, I just need to show that it ...
1
vote
0answers
736 views

Validation error less than training error — implications?

I am running a neural net to predict used car prices, sample size is 800. Using both 10-fold cross validation (10 times) and 1/3 holdback (10 times), the $R^2$ for training is about 0.60 and for ...
1
vote
0answers
134 views

Intraclass correlation to validate a measure

I would like to understand whether the output of an algorithm I developed agrees with the ratings of external observers. The number of observers is 10 and each of them rated on 2 3-point-Likert items ...
0
votes
0answers
7 views

Correcting for multiple data points

I am validating a new measure of therapist competency in psychology. I am getting 4 clients who have taken part in 40 sessions (10 each) to answer a questionnaires about their therapy sessions ...
0
votes
0answers
16 views

validation set with ranking variables

i'm working on an approach of feature selection with SVM model and i have some questions about validation , training and test sets. the idea is to rank variables in decreasing order of relevance ...
0
votes
0answers
14 views

Why use clustering + external measure for validation, instead of supervised learning?

Lets say I have set of genes in individuals and I will use clustering to find groups of individuals with similar genes and then I will check how these groups express itself in form of some disease. I ...
0
votes
0answers
12 views

How do I conduct model validation with glm models that have proportional response variable?

I would like to determine which environmental factors influence turtle nest survival. I ranked glm models with AIC using a step-wise procedure. The response variable is the proportion of eggs that ...
0
votes
0answers
15 views

Alternative to weighted kappa?

I want to compare modeled species occurrence with observed occurrence data. Using confusion matrix and weighed kappa seems to be a good option. In contrast to the normal kappa the weighted kappa takes ...
0
votes
0answers
27 views

Validating formative model - statistical analysis

Based upon literature I have developed a model which I want to validate. Data gathering will be done with the use of surveys. My knowledge of statistics, or more the lack thereof, has raised some ...
0
votes
0answers
49 views

K-fold cross validation for a glmer model with nested data

I'm working on a data set that contains a hierarchical data structure (i.e., GPS locations nested within individual animals). I'm using a generalized linear mixed effects modeling procedure (lme4 ...
0
votes
0answers
26 views

Clustering technique and validation for distance based on file compression

I have a distance matrix based on a normalized compression distance between files: $$ d(x, y) = \frac{ C(xy) - \min \{ C(x), C(y) \} } {\max \{ C(x), C(y) \}} $$ Here, $C(xy)$ is the concatenation ...
0
votes
0answers
10 views

Evaluate and report fit of a model on validation cohort(s)

I trained a random forest regression model M on a training set. I am interested in how well the model predicts the responses in 3 different validation sets. I am also interested in the characteristics ...
0
votes
0answers
14 views

Which phrase is correct? “Out of sample” or “Hold out sample”?

Which one is correct phrase for data that model didn't see? For example i trained my model based on data-set from 200 to 2002. Now i want test my model using 2003 data-set. Which phrase is correct in ...
0
votes
0answers
15 views

How do I compare a new method of statistical sampling with an established one?

Lets say that: There exist 10 countries. I travel to each country and take a random subset of the population of that country and count how many of my subset are woman. I make a table of country to ...
0
votes
0answers
44 views

Testing model fit and running model validation plots for averaged models

I have carried out a model selection process and averaged the models within the top 2 AICcs (using MuMIn in R). I would like to test the model fit using both the chisq and validation/residual plots. ...
0
votes
0answers
88 views

How to test/validate unlabeled data in association rules in R?

I produced association rules by using the arules package (apriori). I'm left with +/- 250 rules. I would like to test/validate the rules that I have, like answering the question: How do I know that ...
0
votes
0answers
24 views

multiple linear model validation

Hello I am a beginner in R. I have a dataset of fish and environmental parameters and anthropogenic pressures from 11 lakes. I have modeled some fish parameters (like fish biomass) as function of ...
0
votes
0answers
35 views

Can the rcorrp.cens paired concordance test be done taking into account strata?

Followup to these discussions: Bootstrapping Hmisc::rcorrp.cens for paired concordance? Stratified concordance index (survival::survConcordance) Is there a way to perform a stratified paired ...
0
votes
0answers
109 views

Implementing Hopkins and Cox-Lewis index in R

I was trying to implement some clustering tendency tools in R, namely the Hopkin's index and the Cox–Lewis index. Here is the link at page 901 to show what they are This is what I managed to come up ...
0
votes
0answers
7 views

ANOVA for 4 groups 2 cohorts?

I have 2 differnt Cohorts from 2 differnt hospitals( unequal number within groups), Each is of 4 classes (differnt complexity) and I have mortaliy figures for each complexiy group. What is the best ...
0
votes
0answers
20 views

Good mean shift clustering datasets

I am working on a modification of the Mean Shift algorithm and would like to validate my clustering results. I am struggling to find suitable clustering datasets to compare performance with. I'm ...
0
votes
0answers
18 views

Can you measure the effectiveness of an estimator using only the recall?

Is it possible to measure the effectiveness of an estimator system Using only the recall measure? Suppose the we have a set of product P(P1, P2 ,... Pn) each product has its ideal reputation value ...
0
votes
0answers
28 views

Methods for ordered logistic regression validation?

We are building now a model using ordered logistic regression in R. I am trying to think one step ahead and am looking into possible approaches for validation. Currently we have like 800 obs ...
0
votes
0answers
8 views

The authenticity of the N-cut measure when the number of components in the data is high

I'm running a clustering task on unlabeled data, and assume we're validating our results by applying the Min-Cut measure as an internal validity index. Let's refer the normalized version of the ...
0
votes
0answers
53 views

Quantifying the predictive ability of a model developed from a huge data set? (variation of bootstrapping?)

I have a statistical model with around 20 predictor variables, built on 90% of a dataset consisting of over 600k observations. The original developer held out 10% of the original dataset for the ...