Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

3
votes
1answer
654 views

Using Adaboost for feature selection?

Is it okay to use Adaboost to do feature selection (selecting a subset of dimensions $S$ from a high-dimensional feature vector $V$)? I divided the samples into four non-overlapping sets: $A$ ...
0
votes
0answers
37 views

How to interpret the results of bootstrapping and Monte Carlo simulation utilised to test lasso logistic regression results?

My situation: sample size: 116 binary outcome (32 events) number predictors: 42 (both continuous and categorical) predictors did not come from the top of my head; their choice was based on the ...
0
votes
0answers
10 views

Fitting Cox Regression / Proportional Hazard Model with x time interaction term in R

I am asking this in the context of wanting to diagnose for violation of proportional hazard assumption and its correction. (Schemper 1992) On p.179 of Hosmer, Lemeshow and May, it says that we can ...
0
votes
0answers
34 views
+50

Ignore strata in external validation of stratified Cox prop hazards model?

I've fit a stratified Cox proportional hazards model to some survival data, where I've stratified by a potential confounder which is the batch the data comes from (there are three batches). Now, I'd ...
4
votes
1answer
670 views

Validate cluster analysis in R

I am trying to validate hierarchical cluster analysis result following a paper by Guy Brock, et al. clValid: An R Package for Cluster Validation (pdf). Do I have to use all these methods? What are the ...
1
vote
3answers
156 views

Performance drop between training and validation datasets

I have been using R's GBM (Gradient Boosting Machine) package for several months. I typically split my data into three partitions: Training, Validation, and Testing. I use the validation data set to ...
3
votes
1answer
35 views

What does a negative Somers' D say about model discriminative power?

I developed a Cox proportional hazard regression model in R. Then I tried using validate in Professor Harrell's rms package to ...
1
vote
1answer
31 views

10-fold cross-validation (high variation)

I am using 10-fold validation method to validate my model. I am using CART model and my sample size $\approx$ 50. Features $\approx$ 9. The 10-fold validated accuracy (averages) is about 76%. However, ...
2
votes
1answer
237 views

Prediction evaluation metric for panel/longitudinal data

I would like to evaluate several different models that provide predictions of behavior at a monthly level. The data is balanced, and $n=$100,000 and $T=$12. The outcome is attending a concert in a ...
3
votes
2answers
863 views

Validity of pseudo-panel data constructed from repeated cross sectional data as a panel data

I am looking at the repeated cross-sectional data from federal reserves, which has both panel data and repeated cross sectional data at different time-points,e.g. 2007-2009 is a panel while 2010 is a ...
0
votes
0answers
10 views

cross-validate hierarchical model for binomial data that is often sparse

I have binomial data (e.g, 130 successes in 4000 trials). In many, if not most, cells of interest, there were few trials and thereby few successes (e.g., 0 successes in 35 trials, 1 successes in 18 ...
3
votes
1answer
107 views

Validation of a questionnaire in a new population

I have 400 responses to a 20 item questionnaire which purports to measure an attitudinal constuct in medical students. The instrument was validated in the US for a single year of medical students and ...
0
votes
0answers
19 views

A way to check the accuracy of a Markov chain?

I posted the same question on MSE since I was not really sure whether to post it here first or not. Anyway since I still did not get any answers I will be posting this here hoping for some help. Say ...
1
vote
1answer
59 views

rms validate on models with a predict function such as coxph and glmnet

I would like to use bootstrapping to evaluate models generated by coxph and glmnet. Would that be somehow possible with rms validate ? From the documentation it seems limited to rms functions (cph, ...
0
votes
0answers
15 views

How to measure classification accuracy based on presence only data?

I have a binomial GLMM which I calibrated with data on recreational visits (presence) compared with random controls where no visits were recorded (absence). I generated the controls myself, whereas ...
0
votes
0answers
8 views

Select final model after many Dev-Val splits

We have a dataset that is quite heterogeneous, and we established that different Dev-Val splits affect the outcome quite bit, even after using stratified sampling. So what we did write a loop around ...
1
vote
1answer
20 views

How to validate a lognormal random walk for time series data

I am currently working on a project where I need to simulate the prices of a set of $D$ substitutable commodities over time. I was hoping to do this using the following $D$-dimensional lognormal ...
0
votes
1answer
78 views

Repeatedly measuring accuracy against the hold out set

I have an iterative document classification task, corpus size = 300,000 documents. The labels are binary valued (yes/no). I wanted to know whether the following methodology is valid. The assumption is ...
2
votes
2answers
3k views

What is the meaning of orthogonal in validation testing?

I have heard the term "orthogonal * validation" used recently. It was used in the context of experimental platform testing. What does this mean? I cannot find anything on it in literature or ...
0
votes
1answer
52 views

How to use k-fold cross validation in naive bayes classifier?

I'm trying to classify text using naive bayes classifier, and also want to use k-fold cross validation to validate the result of classification. But I'm still confused how to use the k-fold cross ...
2
votes
1answer
68 views

Cross-validation for Comparing Clustering Techniques

I'm working on comparing multiple clustering algorithms to each other using the adjusted Rand index for a given dataset. We have a gold standard that we'd like to compare the obtained clustering ...
0
votes
1answer
14 views

what “non-continuous nature of scores” means

1) Can someone explain me the idea of this sentence: "It is noted that nonparametric tests were employed because of the non-continuous nature of the scale scores" (this is related to scale validity ...
0
votes
0answers
7 views

Validation of the gene-signature predictor reconstructed using Cox PH

I reconstructed a prognostic gene model including four genes using univariate and then multivariate Cox PH model. The model is used for predicting prognosis in patients with a specific cancer ...
0
votes
1answer
68 views

Validate a Markov Chain with few states (model switching)

Suppose I have a five state Markov chain. The states are observable (in fact I defined them, they are outcomes of an classification algorithm). So I have a long time series, see the first picture. ...
0
votes
1answer
22 views

How can one evaluate Incremental Clustering Algorithms, in particular the goodness of the clusters formed?

I have been studying an incremental clustering algorithm for a large set of data that exhibit an inherent dynamic behavior (that is new data can get added over time and some older data may get deleted ...
4
votes
1answer
378 views

What is the intuition behind the variation of information (VI) metric for cluster validation?

For non-statisticians like me, it is very difficult to capture the idea of VI metric (variation of information) even after reading the relevant paper by Marina ...
0
votes
0answers
9 views

What is the minimal amount of sample runs required for doing a minimum detection limit?

I have to set up an example of how to do an MDL, and the professor wants to know how many times I'd run the samples. My previous class used as little as 8, but I'm not sure if thats correct or if I ...
1
vote
1answer
134 views

How do you validate your machine learning models?

I am wondering what approaches are commonly used for validating machine learning models designed for classification or prediction tasks: Approaches that am using at the moment: Using truth-sets: - ...
0
votes
1answer
44 views

Validating residual plot count data (different levels)

I am studying the distribution of a marine species using the number of sightings as a dependent variable. When I am trying to validate the plots of the best model I am getting a non-usual pattern, and ...
0
votes
0answers
29 views
0
votes
0answers
43 views

Validating Bootstrapped Probability of Survival Results From Small Sample Size Data

Quite often in industry, due to cost and schedule constraints, decisions must be made on small sample size data. I have 4 cycles-to-failure values resulting from running samples to failure in a ...
0
votes
0answers
14 views

Model validation and verification for Markov Chain switching model

Assuming I have a discrete-time Markov chain with only five states. The chain will be used for the prediction of the macroscopic states which are observable and coming from a timeseries. I use maximum ...
5
votes
1answer
128 views

Validating a logistic regression for a specific $x$

I have a logistic regression model for 0/1 binary response data that is built from samples $(x_1,Y_1),\dots,(x_m,Y_m)$, where $x_1,\dots,x_m$ are, fixed, nonrandom, real values and $Y_1,\dots,Y_m$ are ...
0
votes
1answer
31 views

rms package, getting zero Dxy in validation of cph model

I'm doing Cox proportional hazards analysis using Frank Harrell's rms package (4.2.0), with 3 strata. My model seems reasonably predictive on training data (Dxy=-0.537), but somehow in bootstrap ...
2
votes
0answers
41 views

How to deal with floor effect

I am in the process of validating a five-items scale for measuring dependence on substance 'X'. I have collected data from 98 people who used the substance under consideration at least once weekly for ...
0
votes
0answers
54 views

How to validate goodness of fit and forecasting quality of a model?

I am working on big data sets, which represent electricity consumption on power substation throughout the year. I have data every 10 minutes and multiple (and long) seasonalities. I have a daily, ...
1
vote
1answer
41 views

Out-of-sample vs. test set

Someone asked me if I did out-of-time testing (which I assume is just out-of-sample testing but with a timeline element). But if I have a test set, is that not essentially the same as out-of-sample ...
0
votes
1answer
50 views

How do you measure the accuracy of an inference hypothesis/procedure?

Take inference to mean reasoning/predicting the value of a hidden/laten variable $Z$ given some evidence/data $X$. For example, maybe you are trying to find out if your patient has Cancer (Z = 1 if he ...
0
votes
0answers
89 views

Compare LMM GLMM (generalised linear mixed model, negative binomial) by numerical measure (AIC BIC, cross validation, R² squared) for model validation

How to compare results of generalized linear mixed model (GLMM, negative binomial) with a log transformed linear mixed model (multilevel, hierarchical) . I have a data set (counts), which is nested. ...
0
votes
0answers
25 views

Applying Cox proportional hazards model to new data to get absolute risk (Framingham risk score)

I'm trying to apply one of the Framingham cardiovascular event risk scores to a new dataset in order to get absolute risk. D'Agostino 2013 "Cardiovascular Disease Risk Assessment: Insights from ...
1
vote
0answers
14 views

Derive minimum positive rate of change for co2 data

I have CO2 (in parts per million) data of a closed room. The CO2 data is recorded along with timestamps. Typical difference between two samples is around five minutes. My aim is to find occupancy of ...
2
votes
2answers
42 views

The size of the sample for split validation

At this moment I have a dataset with 4000 samples (50% positive and 50% negative). Normally I would do cross validation for this approach, however besides normal data mining techniques I am also ...
1
vote
3answers
968 views

How to validate a Multinomial Logit and Probit Model fit?

I would like to know how do you determine the performance of your models. That is, if you fit a multinomial logit or probit model for un-ordered discrete choice. What do you use to evaluate whether ...
1
vote
2answers
270 views

Logistic Regression Model Validation

I am validating a logistic regression model. This is the first time i am validating a model. I am using split sampling method. I have split data randomly into two parts - 70% development and 30% ...
0
votes
0answers
18 views

Validating a multivariate categorical model

I assume that my population is a sample of an unknown multivariate categorical distribution $\mathbf{X} = (X_1, X_2, \ldots, X_k)$. From this population, a sample $\mathbf{X^*}$ is available, I assume ...
1
vote
1answer
1k views

Accuracy rate in naive Bayes classification

I am trying to use a naive Bayes classification technique to predict fraudsters (Caller). My training set of 138 instances has 5 columns viz. ...
0
votes
0answers
50 views

Validation - correctly compare and validated imputation models

I've seen a lot of interesting questions here about multiple imputation and also great answers that helped me a lot to impute my data. I've used Predictive Mean Matching, EMB and I would like to use ...
0
votes
1answer
26 views

Computing predicition intervals with cross-validation?

I'm using a k-fold (10-fold) cross-validation while building a model. I'm only using it to get an estimate of the out-of-sample error, not to pick a model from candidates. For example, if I have 30 ...
1
vote
1answer
53 views

Is the statistical significance of a regression meaningful if it has poor out of sample performance?

I want to determine the significance of a particular variable, among many confounders. If I fit a model on the training set and observe a small p value, should I discard the model because it ...
3
votes
2answers
1k views

How do I validate my multiple linear regression model?

I have split my data into two parts. I have used my 80% data to build a multiple regression linear model. Now I want to test it using my rest 20% data. What tools on Minitab do I have to make this ...