Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

5
votes
0answers
152 views

Are world cup predictions testable?

As of today, dozens of soccer world cup predictions exist, some more complex, some more elegant, and most of them predict every nation's "chance" of winning a particular match/ the cup. As I am ...
4
votes
0answers
1k views

Hold out sample vs. cross validation for time series, and how to perform in R

I think out-of-sample validation testing for accuracy is essential in initially judging what time-series forecasts to use. In any case, I've been doing some reading on the two most common methods, ...
3
votes
0answers
171 views

Validating statistical tests for value at risk and expected shortfall

I am trying to figure out if value-at-risk (VaR, a quantile) type tests could capture if expected shortfall (expectations above a quantile) point forecast generated from a type of model could be ...
3
votes
0answers
122 views

Validation of mixed-effect models

I want to use linear mixed effect model for a set of data. After using lme4 package and lmer() function and fitting model, I want to validate my model for other ...
3
votes
0answers
588 views

Logistic Regression Cost Function issue in Matlab

I'm trying to implement a logistic regression function in matlab. I calculated the theta values, linear regression cost function is converging and then I use those parameters in logistic regression ...
2
votes
0answers
49 views

What should I do when logistic regression model doesn't perform well on test sample?

For logistic model, I divided dataset into two parts training sample (70% of 360 data point (observation)) and test sample (rest 30% of 360) randomly. After that I built logistic model on training ...
2
votes
0answers
88 views

Using simulated data to check when patterns in GLMM residual plots are acceptable

I have run the following Poisson GLMM: ...
2
votes
0answers
65 views

How to deal with floor effect

I am in the process of validating a five-items scale for measuring dependence on substance 'X'. I have collected data from 98 people who used the substance under consideration at least once weekly for ...
2
votes
0answers
31 views

Looking for a Good Text on Statistical Analysis of Satisfaction Data

I am looking for a good textbook (or other resource) that covers the analysis of satisfaction data. Most of my data uses likert-type scales. Can anyone recommend something with examples?
2
votes
0answers
141 views

Model validation in Bayesian statistics from a model with latent variables

I am working with some two-regime autoregressive models first introduced by Hamilton in 1989. The specific models is of no great concern to my question, but some variables within my autoregressive ...
2
votes
0answers
157 views

What is the criterion of model validation?

Suppose I have a model that boasts given 1 year's daily data to calibrate the parameters, it could predict the behavior of future 1 month. What should be the criterion for back-testing or, model ...
2
votes
0answers
70 views

Ways to compare ordered binary datasets?

This question regards the best way to compare ordered binary data. My situation is as follows: I'm interested in evaluating how well a model conforms to human performance data on a set of validation ...
2
votes
0answers
125 views

Can holdout validation be systematically biased?

I recently did some experimenting comparing some common method of internal validation. In my field, the use of a single 1:1 holdout validation is extremely common, even with very small datasets, and I ...
2
votes
0answers
83 views

What tasks should I do when performing a model validation?

I have a mixed model built using Data A below, and now I want to validate the model using Data B. What should I do to actually "validate" the model? ...
2
votes
0answers
1k views

Validation error less than training error — implications?

I am running a neural net to predict used car prices, sample size is 800. Using both 10-fold cross validation (10 times) and 1/3 holdback (10 times), the $R^2$ for training is about 0.60 and for ...
2
votes
0answers
157 views

Validation of flow cytometer (Facs Canto II)

I was just wondering if 3 biological samples would be statiscally significant for both precision testing and calibration verification. Precision: to check whether instrument is able to give ...
2
votes
0answers
52 views

Validating previous analysis with new data

I am interested in trying to validate some results from a previous experiment. The experiment involves selecting probes of interest that separate our two populations of interest, e.g. smoker vs. ...
2
votes
0answers
101 views

Validation of data partitioning

I've some numerical sequence of data monitored over time $v(i)$, which belongs to one of two classes, 0 or 1. I monitor the variable several times, obtaining a sequence associated to 0 or 1. I use ...
1
vote
0answers
20 views

When I can say a dataset is imbalanced?

I have a data set with only two outputs: positive and negative The ratio of positive:negative is 3.5:1 In this case, is my data set unbalanced? If so, what metric I should use to report the ...
1
vote
0answers
11 views

Does it make sense to use my in-sample data to look at goodness of fit for my model?

Basically, I came across an article where the authors first ran a logistic regression on a data set to predict the probability (q=demand) of buying their product, as a function of price p and various ...
1
vote
0answers
50 views

Cluster validation method for no cluster labels and differently sized clusters

I'm primarily a programmer and have little to no training in formal maths or statistics of any kind. I'm working on my dissertation (which foolishly is about clustering data), the process is ...
1
vote
0answers
3 views

Validation of prediction interval for count data

I have developed a random-effects (frailty) survival model for repeated events which enables calculating individual-specific mean and prediction interval for the cumulative incidence (rate) of future ...
1
vote
0answers
35 views

How to split training data for parameter tuning and accurate out-of-sample predictions

I am currently working on a kaggle competition; ignoring the details, each training & test set example, in this competition, essentially consists of product and search term pairs. Given each of ...
1
vote
0answers
35 views

Computing c-index for an external validation of a Cox PH model with Stata

I have developed a Cox prognostic model based and internally validated it using bootstrapping. I also evaluated its calibration using Harrell’s $c$ statistic and Somers' $D$. I have also checked ...
1
vote
0answers
26 views

Should difference between accuracy of model on training data and testing data be considered for model selection?

Suppose I have two models (Model 1 and Model 2), Where Accuracy of Model 1 on test data is higher than that in Model 2 Difference between accuracy of model on test data and training data is higher ...
1
vote
0answers
19 views

What does this evaluation function mean?

I am solving a prediction problem where I am supposed to predict the number of clicks, given some historical Data. I am told that my predictions will be evaluated by a normalized weighted mean square ...
1
vote
0answers
22 views

Bootstrap within bootstrap?

I want to use .632 bootstrapping to validate an algorithm. The problem is that the algorithm itself also has bootstrapping procedures. Will that cause bias? Since bootstrapping with replacement will ...
1
vote
0answers
15 views

The burden for verification/validation data

I seek a rigorous or otherwise proof for the need of verification/validation data. Typically it is suggested, after building a model, to do any validation of the model's health on a set of data that ...
1
vote
0answers
123 views

Sample size needed to validate classification/prediction model

Dose any rule of thumb exist (or possible calculation) regarding sample size needed to validate an binary classification model. We have developed this prediction model for a medical condition and ...
1
vote
0answers
658 views

Training/ Test Data with Time Series Model — Forecast with Training Model, or with Model based on Full Data?

Okay, I have a couple books on time series forecasting, but perhaps I need to read a couple more. Here's my question. You want to be able to validate a forecasting model. So you split the data into ...
1
vote
0answers
475 views

Fitting Cox Regression / Proportional Hazard Model with x time interaction term in R

I am asking this in the context of wanting to diagnose for violation of proportional hazard assumption and its correction. (Schemper 1992) On p.179 of Hosmer, Lemeshow and May, it says that we can ...
1
vote
0answers
23 views

Derive minimum positive rate of change for co2 data

I have CO2 (in parts per million) data of a closed room. The CO2 data is recorded along with timestamps. Typical difference between two samples is around five minutes. My aim is to find occupancy of ...
1
vote
0answers
100 views

The effect of oversampling on the positive predictive value

I need to calculate the positive predictive value for a validation set for a rare event. The problem is that the validation set was oversampled for the rare event. The event occurs in 5 percent of the ...
1
vote
0answers
581 views

k-means + linear regression: How to split the data for validation

I want to cluster my data first using k-means and then determine a regression model for each cluster. Then I want to evaluate the performance of this approach using split validation. I can think of ...
1
vote
0answers
97 views

Evaluating Expected Shortfall

I am writing my thesis on VaR and ES risk measurements and have encountered some issues with how to best test the accuracy of ES statistics. My understanding of the topic is that backtesting ES ...
1
vote
0answers
49 views

How to check if the results of two methodologies are the same?

I have two models that essentially are supposed to produce the same results. Given that the first model is correct, what tests are available that I can use to check the validity of the second model ...
1
vote
0answers
605 views

Kendall's tau for holdouts low and not significant - Conjoint

I have done my Conjoint Analysis (fractional factorial design) but when it comes to validating the model, it shows a Kendall's tau for Holdouts of 0.33 and not significant. But Pearson's R and ...
1
vote
0answers
807 views

How to write a loop in R to select multiple regression model and validate it?

I would like to run a loop in R. I have never done this before, so I would be very grateful for your help ! I have a sample set: 25 objects. I would like to draw 1 object from it and use it as a ...
1
vote
0answers
83 views

Assessment of a single predictive model

I am facing a situation where I need to assess the predictive ability of a single (regression) model. I do not need to compare its performance with some other model, I just need to show that it ...
1
vote
0answers
146 views

Intraclass correlation to validate a measure

I would like to understand whether the output of an algorithm I developed agrees with the ratings of external observers. The number of observers is 10 and each of them rated on 2 3-point-Likert items ...
0
votes
0answers
6 views

How to summarize MSE across groups?

I have a large amount of time series data collected for different groups over a 30 year period (dataset x) that can be broken down by sub-group. and corresponding time series data (dataset y) from a ...
0
votes
0answers
30 views

Can I use the Xie-Beni index to validate data transformation parameters in fuzzy c-means clustering?

I am using fuzzy c-means algorithm to cluster my data in various feature spaces and the results differ depending on what kind of transformation I perform on my raw data. I want to know if using the ...
0
votes
0answers
19 views

How to bootstrap validate regression model that involves removing outliers?

Suppose I have the following modelling process: Fit simple linear regression to whole data. Identify outliers, in the sense of having studentized residuals greater than a threshold, and remove them. ...
0
votes
0answers
17 views

External validation of a group's machine learning process

Say there's a group (that is opaque to you) that's heavily using machine learning methods to produce some outputs. You don't have access to their input data or code but can ask high level questions ...
0
votes
0answers
13 views

What sort of cross validation is this?

I've always tested my classification techniques using non-standardised trial and error but I'm interested to see which category my techniques fall under. They seem to fall under several but I'm not ...
0
votes
0answers
12 views

How to find the accuracy of a multiple linear regression model against existing dataset ?`

I need to make a statement of accuracy of my predictive model against existing dataset , how should I proceed ahead with it?
0
votes
0answers
18 views

Calibration test of cox-model in external validation using stata

I'm planning to validate a cox-risk prediction model in external dataset. As a pert of the validation process, I want to test the calibration of my model in the external dataset. I would appreciate if ...
0
votes
0answers
19 views

nltk: odd outputs from bleu_score

For machine translation purposes I use bleu score, which seems to be the validation mechanism of choice (used in the sutskever 2014 sequence-to-sequence). The purpose is to get as high bleu as ...
0
votes
0answers
9 views

dataset construction

I am trying to build a dataset for benchmarking but for which the sizes of each group is quite different. I am feeling there is something a bit dodgy in my current methodology but cannot point it out. ...
0
votes
0answers
16 views

How to determine sample size for validation?

In my study, I have a prediction model built using some previous samples that I have. Now I want to validate this model in an independent data set. Basically, I will use my model to make predictions ...