Tagged Questions

Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing validity of a measurement or instrument -- such as that it measures what it purports to.

152 views

Are world cup predictions testable?

As of today, dozens of soccer world cup predictions exist, some more complex, some more elegant, and most of them predict every nation's "chance" of winning a particular match/ the cup. As I am ...
59 views

How to Validate a Monte Carlo Simulation

I have historical data of a production process, and I've being asked to build a simulation model to predict its performance in the future. Using the historical data, I've being able to obtain the ...
86 views

optimism-corrected regression coefficients using Frank Harrell's method?

I used a regularized (LASSO) cox regression to estimate relapse times of patients and used Frank Harrell's bootstrapping method to obtain an optimism-corrected performance estimate of my model. I am ...
1k views

Hold out sample vs. cross validation for time series, and how to perform in R

I think out-of-sample validation testing for accuracy is essential in initially judging what time-series forecasts to use. In any case, I've been doing some reading on the two most common methods, ...
44 views

Cross-validation scheme used in the Introduction to Statistical Learning, Chapter 6, Lab 3

I've been really enjoying the Introduction to Statistical Learning textbook so far, and I'm currently working my way through chapter 6. I realize that I am very confused by the process used in lab 3 ...
182 views

Validating statistical tests for value at risk and expected shortfall

I am trying to figure out if value-at-risk (VaR, a quantile) type tests could capture if expected shortfall (expectations above a quantile) point forecast generated from a type of model could be ...
123 views

Validation of mixed-effect models

I want to use linear mixed effect model for a set of data. After using lme4 package and lmer() function and fitting model, I want to validate my model for other ...
617 views

Logistic Regression Cost Function issue in Matlab

I'm trying to implement a logistic regression function in matlab. I calculated the theta values, linear regression cost function is converging and then I use those parameters in logistic regression ...
35 views

Choosing evaluation measure for non-parametric clustering

I have to cluster some data using non-parametric clustering technique which is given in this paper. After all the cluster evaluation measure used in this paper is Normalized Mutual Information as they ...
51 views

What should I do when logistic regression model doesn't perform well on test sample?

For logistic model, I divided dataset into two parts training sample (70% of 360 data point (observation)) and test sample (rest 30% of 360) randomly. After that I built logistic model on training ...
97 views

Using simulated data to check when patterns in GLMM residual plots are acceptable

I have run the following Poisson GLMM: ...
73 views

How to deal with floor effect

I am in the process of validating a five-items scale for measuring dependence on substance 'X'. I have collected data from 98 people who used the substance under consideration at least once weekly for ...
31 views

Looking for a Good Text on Statistical Analysis of Satisfaction Data

I am looking for a good textbook (or other resource) that covers the analysis of satisfaction data. Most of my data uses likert-type scales. Can anyone recommend something with examples?
149 views

Model validation in Bayesian statistics from a model with latent variables

I am working with some two-regime autoregressive models first introduced by Hamilton in 1989. The specific models is of no great concern to my question, but some variables within my autoregressive ...
158 views

What is the criterion of model validation?

Suppose I have a model that boasts given 1 year's daily data to calibrate the parameters, it could predict the behavior of future 1 month. What should be the criterion for back-testing or, model ...
71 views

Ways to compare ordered binary datasets?

This question regards the best way to compare ordered binary data. My situation is as follows: I'm interested in evaluating how well a model conforms to human performance data on a set of validation ...
128 views

Can holdout validation be systematically biased?

I recently did some experimenting comparing some common method of internal validation. In my field, the use of a single 1:1 holdout validation is extremely common, even with very small datasets, and I ...
644 views

Kendall's tau for holdouts low and not significant - Conjoint

I have done my Conjoint Analysis (fractional factorial design) but when it comes to validating the model, it shows a Kendall's tau for Holdouts of 0.33 and not significant. But Pearson's R and Kendall'...
87 views

What tasks should I do when performing a model validation?

I have a mixed model built using Data A below, and now I want to validate the model using Data B. What should I do to actually "validate" the model? ...
1k views

Validation error less than training error — implications?

I am running a neural net to predict used car prices, sample size is 800. Using both 10-fold cross validation (10 times) and 1/3 holdback (10 times), the $R^2$ for training is about 0.60 and for ...
160 views

Validation of flow cytometer (Facs Canto II)

I was just wondering if 3 biological samples would be statiscally significant for both precision testing and calibration verification. Precision: to check whether instrument is able to give ...
53 views

Validating previous analysis with new data

I am interested in trying to validate some results from a previous experiment. The experiment involves selecting probes of interest that separate our two populations of interest, e.g. smoker vs. non-...
101 views

Validation of data partitioning

I've some numerical sequence of data monitored over time $v(i)$, which belongs to one of two classes, 0 or 1. I monitor the variable several times, obtaining a sequence associated to 0 or 1. I use ...
11 views

validating a gaussian process fitted to data

I am relatively new to applying Gaussian processes to data. I come from a math background but the most popular literature on it seems to be from a machine learning perspective and not from a ...
12 views

What's the most appropriate way to derive and validate a model with hierarchical data

I am working on a model to predict the risk of some outcomes and could really use some advise: Let's say we have x number of patients, each patient have anywhere between 0 and y number of visits (...
56 views

How to validate a Poisson GLMM model?

I’m using the glmer function from the lme4 package in R to model species richness adjacent ...
25 views

ARIMA model over- or underfitting: compare training and validation performance

I'm doing research using seasonal and nonseasonal ARIMA models. Here's the result of model identification: Based on many sources, Your model is overfitting your training data when you see that ...
28 views

CNN training and overfitting

When testing the training of a CNN code with a small data set (approx 2560 images each for training and validation), what is over-fitting and how can it be mitigated? Arnold
21 views

When I can say a dataset is imbalanced?

I have a data set with only two outputs: positive and negative The ratio of positive:negative is 3.5:1 In this case, is my data set unbalanced? If so, what metric I should use to report the ...
12 views

Does it make sense to use my in-sample data to look at goodness of fit for my model?

Basically, I came across an article where the authors first ran a logistic regression on a data set to predict the probability (q=demand) of buying their product, as a function of price p and various ...
61 views

Cluster validation method for no cluster labels and differently sized clusters

I'm primarily a programmer and have little to no training in formal maths or statistics of any kind. I'm working on my dissertation (which foolishly is about clustering data), the process is ...
4 views

Validation of prediction interval for count data

I have developed a random-effects (frailty) survival model for repeated events which enables calculating individual-specific mean and prediction interval for the cumulative incidence (rate) of future ...
40 views

How to split training data for parameter tuning and accurate out-of-sample predictions

I am currently working on a kaggle competition; ignoring the details, each training & test set example, in this competition, essentially consists of product and search term pairs. Given each of ...
53 views

Computing c-index for an external validation of a Cox PH model with Stata

I have developed a Cox prognostic model based and internally validated it using bootstrapping. I also evaluated its calibration using Harrell’s $c$ statistic and Somers' $D$. I have also checked ...
38 views

Should difference between accuracy of model on training data and testing data be considered for model selection?

Suppose I have two models (Model 1 and Model 2), Where Accuracy of Model 1 on test data is higher than that in Model 2 Difference between accuracy of model on test data and training data is higher ...
19 views

What does this evaluation function mean?

I am solving a prediction problem where I am supposed to predict the number of clicks, given some historical Data. I am told that my predictions will be evaluated by a normalized weighted mean square ...
23 views

Bootstrap within bootstrap?

I want to use .632 bootstrapping to validate an algorithm. The problem is that the algorithm itself also has bootstrapping procedures. Will that cause bias? Since bootstrapping with replacement will ...
16 views

The burden for verification/validation data

I seek a rigorous or otherwise proof for the need of verification/validation data. Typically it is suggested, after building a model, to do any validation of the model's health on a set of data that ...
155 views

Sample size needed to validate classification/prediction model

Dose any rule of thumb exist (or possible calculation) regarding sample size needed to validate an binary classification model. We have developed this prediction model for a medical condition and ...
805 views

Training/ Test Data with Time Series Model — Forecast with Training Model, or with Model based on Full Data?

Okay, I have a couple books on time series forecasting, but perhaps I need to read a couple more. Here's my question. You want to be able to validate a forecasting model. So you split the data into "...
514 views

Fitting Cox Regression / Proportional Hazard Model with x time interaction term in R

I am asking this in the context of wanting to diagnose for violation of proportional hazard assumption and its correction. (Schemper 1992) On p.179 of Hosmer, Lemeshow and May, it says that we can ...
24 views

Derive minimum positive rate of change for co2 data

I have CO2 (in parts per million) data of a closed room. The CO2 data is recorded along with timestamps. Typical difference between two samples is around five minutes. My aim is to find occupancy of ...
635 views

k-means + linear regression: How to split the data for validation

I want to cluster my data first using k-means and then determine a regression model for each cluster. Then I want to evaluate the performance of this approach using split validation. I can think of ...
101 views

Evaluating Expected Shortfall

I am writing my thesis on VaR and ES risk measurements and have encountered some issues with how to best test the accuracy of ES statistics. My understanding of the topic is that backtesting ES ...
51 views

How to check if the results of two methodologies are the same?

I have two models that essentially are supposed to produce the same results. Given that the first model is correct, what tests are available that I can use to check the validity of the second model ...
864 views

How to write a loop in R to select multiple regression model and validate it?

I would like to run a loop in R. I have never done this before, so I would be very grateful for your help ! I have a sample set: 25 objects. I would like to draw 1 object from it and use it as a ...
84 views

Assessment of a single predictive model

I am facing a situation where I need to assess the predictive ability of a single (regression) model. I do not need to compare its performance with some other model, I just need to show that it ...
151 views

Intraclass correlation to validate a measure

I would like to understand whether the output of an algorithm I developed agrees with the ratings of external observers. The number of observers is 10 and each of them rated on 2 3-point-Likert items ...