Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

1
vote
0answers
14 views

Sample size needed to validate classification/prediction model

Dose any rule of thumb exist (or possible calculation) regarding sample size needed to validate an binary classification model. We have developed this prediction model for a medical condition and ...
-1
votes
0answers
16 views

clustering evaluation with mixed data [on hold]

I want to do clustering and evaluating the clusters validity. I have mixed data(mixture of nominal and discrete numerical data). I used gower's coefficient in standard hierarchical clustering and ...
0
votes
1answer
31 views

Model validation using features (R)

I'm trying to create a classification model in R, at the moment i'm using 10 fold CV (using the train function of caret package) ...
0
votes
0answers
19 views

Testing model fit and running model validation plots for averaged models

I have carried out a model selection process and averaged the models within the top 2 AICcs (using MuMIn in R). I would like to test the model fit using both the chisq and validation/residual plots. ...
4
votes
1answer
42 views

Within-group sum of squares of cluster

I have a multivariate dataset for which I have only a table including the cross-wise Euclidean distances between all points and a list giving the assignment of each point to one of several clusters. ...
2
votes
1answer
32 views

Can I apply factor analysis on multiple choice questions?

I am looking to validate a questionnaire and would like to know if I can use factor analysis on the multiple-choice questions (MCQ). Also, I have another section where I am asking about participants ...
1
vote
1answer
78 views

Creating a disease severity score/index

I am trying to design a numerical scale which would describe the severity of a certain disease (in this particular case anaphylaxis). I have a set of clinical symptoms and a database of patients who ...
0
votes
0answers
14 views

How to test/validate unlabeled data in association rules in R?

I produced association rules by using the arules package (apriori). I'm left with +/- 250 rules. I would like to test/validate the rules that I have, like answering the question: How do I know that ...
0
votes
0answers
21 views

multiple linear model validation

Hello I am a beginner in R. I have a dataset of fish and environmental parameters and anthropogenic pressures from 11 lakes. I have modeled some fish parameters (like fish biomass) as function of ...
2
votes
1answer
48 views

Does the position at which maximum distance occurs in a KS test make a difference?

From my understanding of the KS test, fromt the CDF of two datasets, it measures the distance between the two distributions at various points and and compares the 'maximum distance' to a predefined ...
1
vote
1answer
70 views

What is out of time validation in logistic regression model?

I understand out of sample validation very well. Can you explain what is out of time validation? Context A team in my organization has build a churn model for a teleco. Churn rate is 27%. The models ...
6
votes
4answers
120 views

How bad is hyperparameter tuning outside cross-validation?

I know that performing hyperparameter tuning outside of cross-validation can lead to biased-high estimates of external validity, because the dataset that you use to measure performance is the same one ...
1
vote
1answer
44 views

What is the name of this validation procedure?

I have a set of classified data. In order to test the precision of several algorithms- I split the data into train and test sets. For the test set I choose at random 30% of the data and the rest is ...
10
votes
3answers
924 views

Why isn't the holdout method (splitting data into training and testing) used in classical statistics?

In my classroom exposure to data mining, the holdout method was introduced as a way of assessing model performance. However, when I took my first class on linear models, this was not introduced as a ...
0
votes
0answers
30 views

Can the rcorrp.cens paired concordance test be done taking into account strata?

Followup to these discussions: Bootstrapping Hmisc::rcorrp.cens for paired concordance? Stratified concordance index (survival::survConcordance) Is there a way to perform a stratified paired ...
0
votes
0answers
35 views

Implementing Hopkins and Cox-Lewis index in R

I was trying to implement some clustering tendency tools in R, namely the Hopkin's index and the Cox–Lewis index. Here is the link at page 901 to show what they are This is what I managed to come up ...
0
votes
0answers
3 views

ANOVA for 4 groups 2 cohorts?

I have 2 differnt Cohorts from 2 differnt hospitals( unequal number within groups), Each is of 4 classes (differnt complexity) and I have mortaliy figures for each complexiy group. What is the best ...
5
votes
2answers
72 views

How to validate if a sample is independent and identically distributed

How can I check if the data is drawn i.i.d. from an unknown multivariate distribution?
0
votes
0answers
15 views

Good mean shift clustering datasets

I am working on a modification of the Mean Shift algorithm and would like to validate my clustering results. I am struggling to find suitable clustering datasets to compare performance with. I'm ...
0
votes
0answers
14 views

Can you measure the effectiveness of an estimator using only the recall?

Is it possible to measure the effectiveness of an estimator system Using only the recall measure? Suppose the we have a set of product P(P1, P2 ,... Pn) each product has its ideal reputation value ...
0
votes
0answers
20 views

Methods for ordered logistic regression validation?

We are building now a model using ordered logistic regression in R. I am trying to think one step ahead and am looking into possible approaches for validation. Currently we have like 800 obs ...
0
votes
0answers
6 views

The authenticity of the N-cut measure when the number of components in the data is high

I'm running a clustering task on unlabeled data, and assume we're validating our results by applying the Min-Cut measure as an internal validity index. Let's refer the normalized version of the ...
1
vote
1answer
257 views

Classification score for Random Forest

I'm learning about the Decision Tree and Random Forests. But there is something I don't really understand. I have a training set and a cross-validation set. I need to train different Random Forests, ...
1
vote
0answers
107 views

Training/ Test Data with Time Series Model — Forecast with Training Model, or with Model based on Full Data?

Okay, I have a couple books on time series forecasting, but perhaps I need to read a couple more. Here's my question. You want to be able to validate a forecasting model. So you split the data into ...
0
votes
1answer
121 views

R times series — correct use of forecast() and accuracy() in forecast package

Cross-posting this from Stack Overflow, because it's a bit of a stats/ technology cross-over. I'm relatively new to R and the forecast package I believe authored by Rob Hyndman. I'm having trouble ...
1
vote
0answers
223 views

Hold out sample vs. cross validation for time series, and how to perform in R

I think out-of-sample validation testing for accuracy is essential in initially judging what time-series forecasts to use. In any case, I've been doing some reading on the two most common methods, ...
1
vote
1answer
44 views

Variable selection and validation dataset

According to Hastie & Tibshirani, we shouldn't use validation datasets to do variable selection; otherwise, we will overestimate the model fit. However, it seems quite often to select variables ...
0
votes
2answers
49 views

Validating clustering results with labeled data

I am working on a clustering algorithm and would like to validate its performance against a well-known and used dataset: the KDD-CUP 99 dataset ...
0
votes
0answers
51 views

Quantifying the predictive ability of a model developed from a huge data set? (variation of bootstrapping?)

I have a statistical model with around 20 predictor variables, built on 90% of a dataset consisting of over 600k observations. The original developer held out 10% of the original dataset for the ...
0
votes
2answers
33 views

Accepted Method for Selecting a Validation Set in Python

Is there an accepted method for separating out a validation set in python? In R I would use the sample function. I have 4000 training instances as json and I want ...
0
votes
0answers
334 views

How to interpret the results of bootstrapping and Monte Carlo simulation utilised to test lasso logistic regression results?

My situation: sample size: 116 binary outcome (32 events) number predictors: 42 (both continuous and categorical) predictors did not come from the top of my head; their choice was based on the ...
1
vote
0answers
186 views

Fitting Cox Regression / Proportional Hazard Model with x time interaction term in R

I am asking this in the context of wanting to diagnose for violation of proportional hazard assumption and its correction. (Schemper 1992) On p.179 of Hosmer, Lemeshow and May, it says that we can ...
0
votes
1answer
130 views

Ignore strata in external validation of stratified Cox prop hazards model?

I've fit a stratified Cox proportional hazards model to some survival data, where I've stratified by a potential confounder which is the batch the data comes from (there are three batches). Now, I'd ...
3
votes
1answer
105 views

What does a negative Somers' D say about model discriminative power?

I developed a Cox proportional hazard regression model in R. Then I tried using validate in Professor Harrell's rms package to ...
1
vote
1answer
49 views

10-fold cross-validation (high variation)

I am using 10-fold validation method to validate my model. I am using CART model and my sample size $\approx$ 50. Features $\approx$ 9. The 10-fold validated accuracy (averages) is about 76%. However, ...
1
vote
3answers
285 views

Performance drop between training and validation datasets

I have been using R's GBM (Gradient Boosting Machine) package for several months. I typically split my data into three partitions: Training, Validation, and Testing. I use the validation data set to ...
0
votes
0answers
18 views

cross-validate hierarchical model for binomial data that is often sparse

I have binomial data (e.g, 130 successes in 4000 trials). In many, if not most, cells of interest, there were few trials and thereby few successes (e.g., 0 successes in 35 trials, 1 successes in 18 ...
0
votes
0answers
28 views

A way to check the accuracy of a Markov chain?

I posted the same question on MSE since I was not really sure whether to post it here first or not. Anyway since I still did not get any answers I will be posting this here hoping for some help. Say ...
0
votes
0answers
21 views

How to measure classification accuracy based on presence only data?

I have a binomial GLMM which I calibrated with data on recreational visits (presence) compared with random controls where no visits were recorded (absence). I generated the controls myself, whereas ...
0
votes
0answers
12 views

Select final model after many Dev-Val splits

We have a dataset that is quite heterogeneous, and we established that different Dev-Val splits affect the outcome quite bit, even after using stratified sampling. So what we did write a loop around ...
1
vote
1answer
42 views

How to validate a lognormal random walk for time series data

I am currently working on a project where I need to simulate the prices of a set of $D$ substitutable commodities over time. I was hoping to do this using the following $D$-dimensional lognormal ...
0
votes
1answer
534 views

How to use k-fold cross validation in naive bayes classifier?

I'm trying to classify text using naive bayes classifier, and also want to use k-fold cross validation to validate the result of classification. But I'm still confused how to use the k-fold cross ...
0
votes
1answer
15 views

what “non-continuous nature of scores” means

1) Can someone explain me the idea of this sentence: "It is noted that nonparametric tests were employed because of the non-continuous nature of the scale scores" (this is related to scale validity ...
0
votes
0answers
20 views

Validation of the gene-signature predictor reconstructed using Cox PH

I reconstructed a prognostic gene model including four genes using univariate and then multivariate Cox PH model. The model is used for predicting prognosis in patients with a specific cancer ...
0
votes
1answer
109 views

Validate a Markov Chain with few states (model switching)

Suppose I have a five state Markov chain. The states are observable (in fact I defined them, they are outcomes of an classification algorithm). So I have a long time series, see the first picture. ...
0
votes
1answer
38 views

How can one evaluate Incremental Clustering Algorithms, in particular the goodness of the clusters formed?

I have been studying an incremental clustering algorithm for a large set of data that exhibit an inherent dynamic behavior (that is new data can get added over time and some older data may get deleted ...
0
votes
0answers
9 views

What is the minimal amount of sample runs required for doing a minimum detection limit?

I have to set up an example of how to do an MDL, and the professor wants to know how many times I'd run the samples. My previous class used as little as 8, but I'm not sure if thats correct or if I ...
0
votes
0answers
147 views

All neural network designs stop because of early stopping in MATLAB

I'm using patternnet for my binary classification in MATLAB and using ...
0
votes
1answer
103 views

Validating residual plot count data (different levels)

I am studying the distribution of a marine species using the number of sightings as a dependent variable. When I am trying to validate the plots of the best model I am getting a non-usual pattern, and ...
0
votes
0answers
61 views

Validating Bootstrapped Probability of Survival Results From Small Sample Size Data

Quite often in industry, due to cost and schedule constraints, decisions must be made on small sample size data. I have 4 cycles-to-failure values resulting from running samples to failure in a ...