Tagged Questions

Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

1
vote
0answers
10 views

Training/ Test Data with Time Series Model — Forecast with Training Model, or with Model based on Full Data?

Okay, I have a couple books on time series forecasting, but perhaps I need to read a couple more. Here's my question. You want to be able to validate a forecasting model. So you split the data into ...
0
votes
1answer
24 views

R times series — correct use of forecast() and accuracy() in forecast package

Cross-posting this from Stack Overflow, because it's a bit of a stats/ technology cross-over. I'm relatively new to R and the forecast package I believe authored by Rob Hyndman. I'm having trouble ...
1
vote
0answers
22 views

Hold out sample vs. cross validation for time series, and how to perform in R

I think out-of-sample validation testing for accuracy is essential in initially judging what time-series forecasts to use. In any case, I've been doing some reading on the two most common methods, ...
1
vote
0answers
13 views

Variable selection and validation dataset

According to Hastie & Tibshirani, we shouldn't use validation datasets to do variable selection; otherwise, we will overestimate the model fit. However, it seems quite often to select variables ...
0
votes
2answers
33 views

Validating clustering results with labeled data

I am working on a clustering algorithm and would like to validate its performance against a well-known and used dataset: the KDD-CUP 99 dataset ...
0
votes
0answers
40 views

Quantifying the predictive ability of a model developed from a huge data set? (variation of bootstrapping?)

I have a statistical model with around 20 predictor variables, built on 90% of a dataset consisting of over 600k observations. The original developer held out 10% of the original dataset for the ...
0
votes
2answers
21 views

Accepted Method for Selecting a Validation Set in Python

Is there an accepted method for separating out a validation set in python? In R I would use the sample function. I have 4000 training instances as json and I want ...
0
votes
0answers
62 views

How to interpret the results of bootstrapping and Monte Carlo simulation utilised to test lasso logistic regression results?

My situation: sample size: 116 binary outcome (32 events) number predictors: 42 (both continuous and categorical) predictors did not come from the top of my head; their choice was based on the ...
0
votes
0answers
32 views

Fitting Cox Regression / Proportional Hazard Model with x time interaction term in R

I am asking this in the context of wanting to diagnose for violation of proportional hazard assumption and its correction. (Schemper 1992) On p.179 of Hosmer, Lemeshow and May, it says that we can ...
0
votes
1answer
56 views

Ignore strata in external validation of stratified Cox prop hazards model?

I've fit a stratified Cox proportional hazards model to some survival data, where I've stratified by a potential confounder which is the batch the data comes from (there are three batches). Now, I'd ...
3
votes
1answer
50 views

What does a negative Somers' D say about model discriminative power?

I developed a Cox proportional hazard regression model in R. Then I tried using validate in Professor Harrell's rms package to ...
1
vote
1answer
38 views

10-fold cross-validation (high variation)

I am using 10-fold validation method to validate my model. I am using CART model and my sample size $\approx$ 50. Features $\approx$ 9. The 10-fold validated accuracy (averages) is about 76%. However, ...
1
vote
3answers
188 views

Performance drop between training and validation datasets

I have been using R's GBM (Gradient Boosting Machine) package for several months. I typically split my data into three partitions: Training, Validation, and Testing. I use the validation data set to ...
0
votes
0answers
15 views

cross-validate hierarchical model for binomial data that is often sparse

I have binomial data (e.g, 130 successes in 4000 trials). In many, if not most, cells of interest, there were few trials and thereby few successes (e.g., 0 successes in 35 trials, 1 successes in 18 ...
0
votes
0answers
20 views

A way to check the accuracy of a Markov chain?

I posted the same question on MSE since I was not really sure whether to post it here first or not. Anyway since I still did not get any answers I will be posting this here hoping for some help. Say ...
0
votes
0answers
16 views

How to measure classification accuracy based on presence only data?

I have a binomial GLMM which I calibrated with data on recreational visits (presence) compared with random controls where no visits were recorded (absence). I generated the controls myself, whereas ...
0
votes
0answers
9 views

Select final model after many Dev-Val splits

We have a dataset that is quite heterogeneous, and we established that different Dev-Val splits affect the outcome quite bit, even after using stratified sampling. So what we did write a loop around ...
1
vote
1answer
25 views

How to validate a lognormal random walk for time series data

I am currently working on a project where I need to simulate the prices of a set of $D$ substitutable commodities over time. I was hoping to do this using the following $D$-dimensional lognormal ...
0
votes
1answer
106 views

How to use k-fold cross validation in naive bayes classifier?

I'm trying to classify text using naive bayes classifier, and also want to use k-fold cross validation to validate the result of classification. But I'm still confused how to use the k-fold cross ...
0
votes
1answer
15 views

what “non-continuous nature of scores” means

1) Can someone explain me the idea of this sentence: "It is noted that nonparametric tests were employed because of the non-continuous nature of the scale scores" (this is related to scale validity ...
0
votes
0answers
9 views

Validation of the gene-signature predictor reconstructed using Cox PH

I reconstructed a prognostic gene model including four genes using univariate and then multivariate Cox PH model. The model is used for predicting prognosis in patients with a specific cancer ...
0
votes
1answer
76 views

Validate a Markov Chain with few states (model switching)

Suppose I have a five state Markov chain. The states are observable (in fact I defined them, they are outcomes of an classification algorithm). So I have a long time series, see the first picture. ...
0
votes
1answer
23 views

How can one evaluate Incremental Clustering Algorithms, in particular the goodness of the clusters formed?

I have been studying an incremental clustering algorithm for a large set of data that exhibit an inherent dynamic behavior (that is new data can get added over time and some older data may get deleted ...
0
votes
0answers
9 views

What is the minimal amount of sample runs required for doing a minimum detection limit?

I have to set up an example of how to do an MDL, and the professor wants to know how many times I'd run the samples. My previous class used as little as 8, but I'm not sure if thats correct or if I ...
0
votes
0answers
48 views

All neural network designs stop because of early stopping in MATLAB

I'm using patternnet for my binary classification in MATLAB and using ...
0
votes
1answer
57 views

Validating residual plot count data (different levels)

I am studying the distribution of a marine species using the number of sightings as a dependent variable. When I am trying to validate the plots of the best model I am getting a non-usual pattern, and ...
0
votes
0answers
50 views

Validating Bootstrapped Probability of Survival Results From Small Sample Size Data

Quite often in industry, due to cost and schedule constraints, decisions must be made on small sample size data. I have 4 cycles-to-failure values resulting from running samples to failure in a ...
0
votes
0answers
14 views

Model validation and verification for Markov Chain switching model

Assuming I have a discrete-time Markov chain with only five states. The chain will be used for the prediction of the macroscopic states which are observable and coming from a timeseries. I use maximum ...
0
votes
1answer
39 views

rms package, getting zero Dxy in validation of cph model

I'm doing Cox proportional hazards analysis using Frank Harrell's rms package (4.2.0), with 3 strata. My model seems reasonably predictive on training data (Dxy=-0.537), but somehow in bootstrap ...
0
votes
0answers
68 views

How to validate goodness of fit and forecasting quality of a model?

I am working on big data sets, which represent electricity consumption on power substation throughout the year. I have data every 10 minutes and multiple (and long) seasonalities. I have a daily, ...
1
vote
1answer
53 views

Out-of-sample vs. test set

Someone asked me if I did out-of-time testing (which I assume is just out-of-sample testing but with a timeline element). But if I have a test set, is that not essentially the same as out-of-sample ...
2
votes
0answers
43 views

How to deal with floor effect

I am in the process of validating a five-items scale for measuring dependence on substance 'X'. I have collected data from 98 people who used the substance under consideration at least once weekly for ...
0
votes
1answer
59 views

How do you measure the accuracy of an inference hypothesis/procedure?

Take inference to mean reasoning/predicting the value of a hidden/laten variable $Z$ given some evidence/data $X$. For example, maybe you are trying to find out if your patient has Cancer (Z = 1 if he ...
0
votes
0answers
110 views

Compare LMM GLMM (generalised linear mixed model, negative binomial) by numerical measure (AIC BIC, cross validation, R² squared) for model validation

How to compare results of generalized linear mixed model (GLMM, negative binomial) with a log transformed linear mixed model (multilevel, hierarchical) . I have a data set (counts), which is nested. ...
0
votes
0answers
34 views

Applying Cox proportional hazards model to new data to get absolute risk (Framingham risk score)

I'm trying to apply one of the Framingham cardiovascular event risk scores to a new dataset in order to get absolute risk. D'Agostino 2013 "Cardiovascular Disease Risk Assessment: Insights from ...
1
vote
1answer
144 views

How do you validate your machine learning models?

I am wondering what approaches are commonly used for validating machine learning models designed for classification or prediction tasks: Approaches that am using at the moment: Using truth-sets: - ...
1
vote
0answers
16 views

Derive minimum positive rate of change for co2 data

I have CO2 (in parts per million) data of a closed room. The CO2 data is recorded along with timestamps. Typical difference between two samples is around five minutes. My aim is to find occupancy of ...
2
votes
2answers
47 views

The size of the sample for split validation

At this moment I have a dataset with 4000 samples (50% positive and 50% negative). Normally I would do cross validation for this approach, however besides normal data mining techniques I am also ...
1
vote
2answers
417 views

Logistic Regression Model Validation

I am validating a logistic regression model. This is the first time i am validating a model. I am using split sampling method. I have split data randomly into two parts - 70% development and 30% ...
0
votes
0answers
20 views

Validating a multivariate categorical model

I assume that my population is a sample of an unknown multivariate categorical distribution $\mathbf{X} = (X_1, X_2, \ldots, X_k)$. From this population, a sample $\mathbf{X^*}$ is available, I assume ...
1
vote
1answer
60 views

Is the statistical significance of a regression meaningful if it has poor out of sample performance?

I want to determine the significance of a particular variable, among many confounders. If I fit a model on the training set and observe a small p value, should I discard the model because it ...
0
votes
0answers
52 views

Validation - correctly compare and validated imputation models

I've seen a lot of interesting questions here about multiple imputation and also great answers that helped me a lot to impute my data. I've used Predictive Mean Matching, EMB and I would like to use ...
4
votes
0answers
129 views

Are world cup predictions testable?

As of today, dozens of soccer world cup predictions exist, some more complex, some more elegant, and most of them predict every nation's "chance" of winning a particular match/ the cup. As I am ...
4
votes
0answers
207 views

Internal validation via bootstrap: What ROC curve to present?

I am using the bootstrap approach for internal validation of a multivariate model built with either standard logistic regression OR elastic net. The procedure I use is as follows: 1) build model ...
0
votes
1answer
29 views

Computing predicition intervals with cross-validation?

I'm using a k-fold (10-fold) cross-validation while building a model. I'm only using it to get an estimate of the out-of-sample error, not to pick a model from candidates. For example, if I have 30 ...
2
votes
2answers
71 views

Split train//validation/test sets by time, is it correct?

Here's the scenario, slightly altered to a common one. Credit card fraud, payments for the last 12 months (a rolling window). Train with the data from the first 10 months, validate with data from the ...
-1
votes
1answer
14 views

On which data should the lift be calculated i.e. Training set or Test set and why?

On which data should the lift be calculated i.e. Training set or Test set and why? What does the lift value 115% mean
0
votes
0answers
46 views

How would you validate a random walk model?

I have used a random walk model and Gibbs sampling (more specifically RJAGS) in order to obtain posterior of the state given the observations. In this case the state is the true proportion of the ...
0
votes
1answer
36 views

use of validation set on lasso cross validation

When training a model a train, a validation and test set are used. I was wondering if there is any paper or example that proves that the use of an independent validation set increase the performance ...
0
votes
0answers
27 views

Incorporating validation data into training set

Suppose that I divide my data for modelling purposes into training, testing, and validation which will then be deployed for an application (as in the response to this question). Why not incorporate ...