Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

1
vote
0answers
371 views

Kendall's tau for holdouts low and not significant - Conjoint

I have done my Conjoint Analysis (fractional factorial design) but when it comes to validating the model, it shows a Kendall's tau for Holdouts of 0.33 and not significant. But Pearson's R and ...
3
votes
1answer
132 views

Statistical measures for data validation

I have to validate the data in one database by comparing it to data from a validated application system. The solution I came up with was (at least in theory) creating a statistical method that would ...
4
votes
1answer
851 views

Validate cluster analysis in R

I am trying to validate hierarchical cluster analysis result following a paper by Guy Brock, et al. clValid: An R Package for Cluster Validation (pdf). Do I have to use all these methods? What are the ...
1
vote
1answer
314 views

.632+ bootstrap estimator-defining gamma in continuous variable case

I am trying to implement the .632+ bootstrap estimator for internal validation as proposed by Efron and Tibshiraini 1997. Looking at the paper, I can see how gamma is defined in the case of ...
3
votes
1answer
295 views

Determining values of correction factor based on x bins in observed vs. actual data

I am trying to automate a problem I usually solve by hand. I have a sensor that collects data from the field. Every 6 months or so, I have to do a calibration on that sensor by collecting ...
11
votes
1answer
110 views

When building a regression model using separate modeling/validation sets, is it appropriate to “recirculate” the validation data?

Suppose I've got an 80/20 split between modeling/validation observations. I've fit a model to the modeling data set, and I'm comfortable with the error that I'm seeing on the validation data set. ...
3
votes
2answers
2k views

How do I validate my multiple linear regression model?

I have split my data into two parts. I have used my 80% data to build a multiple regression linear model. Now I want to test it using my rest 20% data. What tools on Minitab do I have to make this ...
0
votes
0answers
426 views

How to write a loop in R to select multiple regression model and validate it?

I would like to run a loop in R. I have never done this before, so I would be very grateful for your help ! I have a sample set: 25 objects. I would like to draw 1 object from it and use it as a ...
2
votes
1answer
267 views

Prediction evaluation metric for panel/longitudinal data

I would like to evaluate several different models that provide predictions of behavior at a monthly level. The data is balanced, and $n=$100,000 and $T=$12. The outcome is attending a concert in a ...
0
votes
2answers
139 views

Linear SVM C optimisation data, how to partition into train, model construction, test

My aim is to find the best C for a linear SVM classification using libsvm, I have 120 instances in total and 2 classes which I want to classify. I have question regarding partitioning a the dataset ...
3
votes
1answer
119 views

Should my test set be balanced or imbalanced?

I have an imbalanced dataset (90% class 0 10% class 1), should I first split it in training and test set, then balance my training set (my test set would still be imbalanced), or could I randomly ...
2
votes
0answers
74 views

What tasks should I do when performing a model validation?

I have a mixed model built using Data A below, and now I want to validate the model using Data B. What should I do to actually "validate" the model? ...
2
votes
2answers
94 views

Validate a medical test

I've never done something like this before, thus I do not even know where to start with calculating whatever I need. Following scenario: We measure a medical value (Glucose) with our device, and ...
3
votes
1answer
197 views

Validation of a scale for a different population (CFA)

I'm currently adapting a coping questionnaire which has been widely used in sports to be used with a different population (musicians), so I need to address its reliability and validity with this new ...
3
votes
1answer
780 views

Using Adaboost for feature selection?

Is it okay to use Adaboost to do feature selection (selecting a subset of dimensions $S$ from a high-dimensional feature vector $V$)? I divided the samples into four non-overlapping sets: $A$ ...
3
votes
1answer
58 views

Mixing User Data For Cross-Validation

I have data from 12 users, and want to perform cross-validation. Is it necessary that I create my training and test data from different users, or can all of the data be randomized, and then split into ...
1
vote
0answers
76 views

Assessment of a single predictive model

I am facing a situation where I need to assess the predictive ability of a single (regression) model. I do not need to compare its performance with some other model, I just need to show that it ...
-1
votes
1answer
77 views

validating a regression model

I want to build a regression model to predict daily income from customers. I have 2 problems: 1. Choosing data for the training set - do I use daily income from 1 month ago, 6 month ago etc. 2. How do ...
0
votes
1answer
154 views

Using components from PCA as index / measurement?

I have used Principal Component Analysis to find main components that desribe the perceptions of the environment. The goal is to use those main components as index / measurement for another study. My ...
2
votes
2answers
2k views

Computing c-index for an external validation of a Cox PH model with R

First off, I'll state that I'm aware many questions get asked about the c-index. I've searched this site and others, and I haven't found an answer for my situation. I can successfully use ...
4
votes
2answers
476 views

In logistic regression, does the lack of significance of the parameter estimates in a test sample indicate overfitting?

I am trying to build a logistic regression model where I have a dependent variable $y$ and independent variables $x_1$, $x_2$... $x_n$. $y$ can take only two values - 0 or 1. My original modelling ...
4
votes
1answer
442 views

Best method to validate a multiply imputed Cox model with R?

This question is with regards to using a test data set to validate an imputed Cox model using R. With a non-imputed data set I would use val.surv() from ...
1
vote
0answers
582 views

Validation error less than training error — implications?

I am running a neural net to predict used car prices, sample size is 800. Using both 10-fold cross validation (10 times) and 1/3 holdback (10 times), the $R^2$ for training is about 0.60 and for ...
3
votes
0answers
365 views

Logistic Regression Cost Function issue in Matlab

I'm trying to implement a logistic regression function in matlab. I calculated the theta values, linear regression cost function is converging and then I use those parameters in logistic regression ...
2
votes
1answer
201 views

What is the best way to compute classifier performance metrics given a confusion matrix?

Let's say I compute a confusion matrix, in the sense defined here: http://www.gabormelli.com/RKB/Confusion_Matrix I could easily compute the number of True Negatives (TN), True Positives (TP), False ...
2
votes
0answers
129 views

Validation of flow cytometer (Facs Canto II)

I was just wondering if 3 biological samples would be statiscally significant for both precision testing and calibration verification. Precision: to check whether instrument is able to give ...
5
votes
1answer
130 views

Validating a logistic regression for a specific $x$

I have a logistic regression model for 0/1 binary response data that is built from samples $(x_1,Y_1),\dots,(x_m,Y_m)$, where $x_1,\dots,x_m$ are, fixed, nonrandom, real values and $Y_1,\dots,Y_m$ are ...
1
vote
1answer
599 views

Validation of logistic regression - goodness of fit (pearson)

I have developed a scoring system using logistic regression. The score ranges between 0 and 6 (using integers) and predicts death. It does not use a conventional regression formula and thus I am not ...
2
votes
1answer
92 views

Is a different CV arrangement the same as a validation set?

I have a smallish dataset ~ 1500 rows X 500 columns. I've been using a standard 5 fold CV setup where row 1 = CV set1, row2 = CV set2, ... row 6 = CV set1,etc. I'm at the point where I'm trying to ...
2
votes
1answer
63 views

Resources for developing prognostic index-scores

I am looking for online resources or books explaining how to develop prognostic indexes-scores step by step. I am mainly wondering about model validation and transformation of regression coefficients ...
4
votes
1answer
2k views

Diagnostic plots for lmer

I am trying to produce a glmm using the lme4 package in R. To validate my model I would like to produce some diagnostic plots ...
3
votes
2answers
960 views

Validity of pseudo-panel data constructed from repeated cross sectional data as a panel data

I am looking at the repeated cross-sectional data from federal reserves, which has both panel data and repeated cross sectional data at different time-points,e.g. 2007-2009 is a panel while 2010 is a ...
2
votes
3answers
3k views

What is the meaning of orthogonal in validation testing?

I have heard the term "orthogonal * validation" used recently. It was used in the context of experimental platform testing. What does this mean? I cannot find anything on it in literature or ...
2
votes
2answers
924 views

What are acceptable validation or cross validation error rates?

Is there a commonly acceptable error rate for validation? As in, if the error rate is less than X %, then my machine learning method would be considered "successful". I'm looking for something ...
1
vote
0answers
127 views

Intraclass correlation to validate a measure

I would like to understand whether the output of an algorithm I developed agrees with the ratings of external observers. The number of observers is 10 and each of them rated on 2 3-point-Likert items ...
2
votes
1answer
730 views

Validation of a linear regression model using R

I have created a multiple linear regression model with R using lm and glm. I am using lm on ...
0
votes
0answers
597 views

Method to output statistical score when validating using bootstrap method on ordinal logistic regression model

It is about sas programming. Thank you in advance. I build an ordinal logistic regression model on my data. ...
1
vote
1answer
783 views

How to select validation data when training a neural network?

I am training a neural network with time dependent financial data. In order to avoid overfitting I would like to stop the training at the point where my neural network stops improving on a set of ...
2
votes
1answer
80 views

Bayesian $\chi^2$ discrepancy

We define the $\chi^2$ discrepancy as $$T(y,\theta)=\sum_i\frac{(y_i-E(y_i|\theta))^2}{\mathrm{var}(y_i|\theta)}$$ (definition from Gelman). How do we apply this if $\theta$ is a random variable, and ...
1
vote
0answers
110 views

Validation of Bayesian hierarchical model [duplicate]

Possible Duplicate: Validation techniques for hierarchical model I already posted this to Math.SE, but I realized this would probably be a better place to post to. I have a hierarchical ...
9
votes
2answers
2k views

What is the procedure for “bootstrap validation” (a.k.a. “resampling cross-validation”)?

"Bootstrap validation"/"resampling cross-validation" is new to me, but was discussed by the answer to this question. I gather it involves 2 types of data: the real data and simulated data, where a ...
4
votes
1answer
3k views

Model validation after fitting a negative binomial GLM in R

Ok, I have searched and searched and just have no clue where to start. First, what I would like to do is produce a QQ-plot (or even a readable residual plot) to look at the fit of my model. I guess ...
2
votes
3answers
835 views

Back propagation neural network data input advice

I am currently doing a project which involves pothole detection and neural networks. So far, I have an Android phone that reads Accelerometer readings and writes the X,Y,Z Axis aswell as the Amplitude ...
2
votes
4answers
605 views

Statistical tests to do data quality checks

I want to identify bad responses in the data we get from our vendors. To assess the quality of the data, what statistical tests can i apply to identify bad respondents? I performed Discriminant ...
27
votes
2answers
3k views

How to draw valid conclusions from “big data”?

"Big data" is everywhere in the media. Everybody says that "big data" is the big thing for 2012, e.g. KDNuggets poll on hot topics for 2012. However, I have deep concerns here. With big data, ...
2
votes
0answers
50 views

Validating previous analysis with new data

I am interested in trying to validate some results from a previous experiment. The experiment involves selecting probes of interest that separate our two populations of interest, e.g. smoker vs. ...
0
votes
1answer
314 views

What is a good academic citation for cross-validation?

I performed a holdout cross-validation analysis on a multilevel model fit. The purpose of this was to show that we didn't have a problem with over-fitting, for which it worked just fine. Now we are ...
5
votes
1answer
118 views

What do you do with your testing data?

Suppose you perform a parameter selection study in an environment with not-so-many data points. You divide the available data into training and testing set and you plan to validate the future model on ...
5
votes
1answer
295 views

Name of mean absolute error analogue to Brier score?

Yesterday's question Determine accuracy of model which estimates probability of event got me curious about probability scoring. The Brier score $$\frac{1}{N}\sum\limits _{i=1}^{N}(prediction_i - ...
3
votes
2answers
466 views

Logistic regression performs better on validation data

Recently I've been building a model using logistic regression. To my suprisise LIFT chart looks better on the validation data than on the training data, the same is with ROC. All variables in the ...