The validation tag has no wiki summary.
19
votes
4answers
410 views
As a reviewer, can I justify requesting data and code be made available even if the journal does not?
As science must be reproducible, by definition, there is increasing recognition that data and code are an essential component of the reproduciblity, as discussed by the Yale Roundtable for data and ...
16
votes
2answers
1k views
How to draw valid conclusions from “big data”?
"Big data" is everywhere in the media. Everybody says that "big data" is the big thing for 2012, e.g. KDNuggets poll on hot topics for 2012. However, I have deep concerns here. With big data, ...
12
votes
3answers
7k views
What is the difference between test set and validation set?
I found this confusing when I use the neural network toolbox in Matlab.
It divided the raw data set into three parts:
training set
validation set
test set
I notice in many training or learning ...
6
votes
2answers
636 views
What is the procedure for “bootstrap validation” (a.k.a. “resampling cross-validation”)?
"Bootstrap validation"/"resampling cross-validation" is new to me, but was discussed by the answer to this question. I gather it involves 2 types of data: the real data and simulated data, where a ...
5
votes
1answer
97 views
What do you do with your testing data?
Suppose you perform a parameter selection study in an environment with not-so-many data points. You divide the available data into training and testing set and you plan to validate the future model on ...
5
votes
2answers
3k views
How to make representative sample set from a large overall dataset?
What are the statistical techniques to create a sample set, which is representative of the entire population (with a known confidence level)?
Also,
How to validate, if the sample fits the overall ...
4
votes
2answers
218 views
In logistic regression, does the lack of significance of the parameter estimates in a test sample indicate overfitting?
I am trying to build a logistic regression model where I have a dependent variable $y$ and independent variables $x_1$, $x_2$... $x_n$. $y$ can take only two values - 0 or 1.
My original modelling ...
4
votes
2answers
788 views
Variance explained of a mixed effects model in a new data set
I'm using the following function to calculate Edwards R^2 (formula 19 in Edwards et al. 2008) of a mixed effects model (I hope the implementation is correct):
...
4
votes
1answer
855 views
Diagnostic plots for lmer
I am trying to produce a glmm using the lme4 package in R. To validate my model I would like to produce some diagnostic plots ...
4
votes
1answer
638 views
Best practices for measuring and avoiding overfitting?
I am developing automated trading systems for the stock market. The big challenge has been overfitting. Can your recommend some resources describing methods for measuring and avoiding overfitting?
I ...
4
votes
1answer
259 views
Compare modeled (fitted) paired data to actual data in forecasting problem (Excel sheet included)
In my endeavor to learn forecasting and improve my statistical knowledge, I've decided to forecast the population of a certain area. I've attached the excel file that I used. I know I'm using excel ...
3
votes
2answers
312 views
Logistic regression performs better on validation data
Recently I've been building a model using logistic regression. To my suprisise LIFT chart looks better on the validation data than on the training data, the same is with ROC. All variables in the ...
3
votes
1answer
1k views
Model validation after fitting a negative binomial GLM in R
Ok, I have searched and searched and just have no clue where to start. First, what I would like to do is produce a QQ-plot (or even a readable residual plot) to look at the fit of my model. I guess ...
3
votes
1answer
146 views
Name of mean absolute error analogue to Brier score?
Yesterday's question Determine accuracy of model which estimates probability of event got me curious about probability scoring.
The Brier score
$$\frac{1}{N}\sum\limits _{i=1}^{N}(prediction_i - ...
3
votes
1answer
50 views
Mixing User Data For Cross-Validation
I have data from 12 users, and want to perform cross-validation. Is it necessary that I create my training and test data from different users, or can all of the data be randomized, and then split into ...
3
votes
1answer
478 views
How to choose training and test sets
I would like to propose a single model (decision tree), that is very variable, and validate it. I have choosen parameters after I had obtained good quality measures with a cross-validation.
I could ...
3
votes
1answer
56 views
Validation of a scale for a different population (CFA)
I'm currently adapting a coping questionnaire which has been widely used in sports to be used with a different population (musicians), so I need to address its reliability and validity with this new ...
3
votes
0answers
193 views
Logistic Regression Cost Function issue in Matlab
I'm trying to implement a logistic regression function in matlab. I calculated the theta values, linear regression cost function is converging and then I use those parameters in logistic regression ...
3
votes
0answers
76 views
Validating a logistic regression for a specific $x$
I have a logistic regression model for 0/1 binary response data that is built from samples $(x_1,Y_1),\ldots,(x_m,Y_m)$, where $x_1,\ldots,x_m$ are, fixed, nonrandom, real values and $Y_1,\ldots,Y_m$ ...
2
votes
1answer
68 views
Is a different CV arrangement the same as a validation set?
I have a smallish dataset ~ 1500 rows X 500 columns. I've been using a standard 5 fold CV setup where row 1 = CV set1, row2 = CV set2, ... row 6 = CV set1,etc.
I'm at the point where I'm trying to ...
2
votes
2answers
67 views
Validate a medical test
I've never done something like this before, thus I do not even know where to start with calculating whatever I need.
Following scenario:
We measure a medical value (Glucose) with our device, and ...
2
votes
1answer
174 views
Validating a paper questionnaire in a web-based format
I would like to validate a commonly used questionnaire for internet-based administration. What statistical tests should I be using?
2
votes
2answers
279 views
What are acceptable validation or cross validation error rates?
Is there a commonly acceptable error rate for validation? As in, if the error rate is less than X %, then my machine learning method would be considered "successful".
I'm looking for something ...
2
votes
4answers
335 views
Statistical tests to do data quality checks
I want to identify bad responses in the data we get from our vendors. To assess the quality of the data, what statistical tests can i apply to identify bad respondents?
I performed Discriminant ...
2
votes
1answer
572 views
Weight variables for predictive model
I received a question today that I wasn't exactly sure how to answer.
I have built a predictive model using a fairly basic logistic regression that works pretty well and fits our business needs. ...
2
votes
1answer
287 views
Validity of pseudo-panel data constructed from repeated cross sectional data as a panel data
I am looking at the repeated cross-sectional data from federal reserves, which has both panel data and repeated cross sectional data at different time-points,e.g. 2007-2009 is a panel while 2010 is a ...
2
votes
1answer
150 views
Best method to validate a multiply imputed Cox model with R?
This question is with regards to using a test data set to validate an imputed Cox model using R. With a non-imputed data set I would use val.surv() from ...
2
votes
1answer
62 views
What is the best way to compute classifier performance metrics given a confusion matrix?
Let's say I compute a confusion matrix, in the sense defined here: http://www.gabormelli.com/RKB/Confusion_Matrix
I could easily compute the number of True Negatives (TN), True Positives (TP), False ...
2
votes
1answer
41 views
Resources for developing prognostic index-scores
I am looking for online resources or books explaining how to develop prognostic indexes-scores step by step. I am mainly wondering about model validation and transformation of regression coefficients ...
2
votes
1answer
549 views
What is the meaning of orthogonal in validation testing?
I have heard the term "orthogonal * validation" used recently. It was used in the context of experimental platform testing. What does this mean? I cannot find anything on it in literature or ...
2
votes
1answer
173 views
What resources/methods exist for testing/validation or evaluation of Statistical Methods
I'm interested in the process of testing or validating a particular implementation of a statistical method, and what datasets and/or published analysis exist that could be used to do this in practice.
...
2
votes
0answers
56 views
What tasks should I do when performing a model validation?
I have a mixed model built using Data A below, and now I want to validate the model using Data B. What should I do to actually "validate" the model?
...
2
votes
0answers
44 views
Validation of flow cytometer (Facs Canto II)
I was just wondering if 3 biological samples would be statiscally significant for both precision testing and calibration verification.
Precision: to check whether instrument is able to give ...
2
votes
1answer
307 views
Validation of a linear regression model using R
I have created a multiple linear regression model with R using lm and glm. I am using lm on ...
2
votes
3answers
534 views
Back propagation neural network data input advice
I am currently doing a project which involves pothole detection and neural networks. So far, I have an Android phone that reads Accelerometer readings and writes the X,Y,Z Axis aswell as the Amplitude ...
2
votes
0answers
45 views
Validating previous analysis with new data
I am interested in trying to validate some results from a previous experiment. The experiment involves selecting probes of interest that separate our two populations of interest, e.g. smoker vs. ...
2
votes
0answers
79 views
Validation of data partitioning
I've some numerical sequence of data monitored over time $v(i)$, which belongs to one of two classes, 0 or 1. I monitor the variable several times, obtaining a sequence associated to 0 or 1.
I use ...
1
vote
2answers
917 views
What is a consistency check?
I was asked such a question as "Did you do any consistency check in your daily work?" during a phone interview for a Biostatistician position. I don't know what to answer. Any information is ...
1
vote
1answer
485 views
How to plot AIC values when using the leaps package?
Does anybody know how to plot all AIC values for different size models, when using the command regsubsets from the package ...
1
vote
1answer
354 views
Validation of logistic regression - goodness of fit (pearson)
I have developed a scoring system using logistic regression. The score ranges between 0 and 6 (using integers) and predicts death. It does not use a conventional regression formula and thus I am not ...
1
vote
1answer
67 views
Bayesian $\chi^2$ discrepancy
We define the $\chi^2$ discrepancy as $$T(y,\theta)=\sum_i\frac{(y_i-E(y_i|\theta))^2}{\mathrm{var}(y_i|\theta)}$$
(definition from Gelman). How do we apply this if $\theta$ is a random variable, and ...
1
vote
1answer
276 views
Why are most standard goodness of fit tests based only on continuous distributions?
I tried to search info regarding this fact but I don't really understand why most of the standard goodness of fit tests (e.g. Kolmogorov-Smirnov, Anderson-Darling, a part of the Chi-square test, ...
1
vote
2answers
171 views
Computing c-index for an external validation of a Cox PH model with R
First off, I'll state that I'm aware many questions get asked about the c-index. I've searched this site and others, and I haven't found an answer for my situation. I can successfully use ...
1
vote
1answer
91 views
Difference between Step validation and Cross validation
I'm trying to model some chemical data from a series of spectra. I've gone through the preprocessing step and selected "autoscale" (I'm using Pirouette). And when it comes to validation I'm a little ...
1
vote
1answer
237 views
Validating a predicted standard deviation
I would like to validate the standard deviation which a model reports, because I think it is underestimated, because in the procedure some model selection is involved.
After running the model on a ...
1
vote
1answer
145 views
Using Adaboost for feature selection?
Is it okay to use Adaboost to do feature selection (selecting a subset of dimensions $S$ from a high-dimensional feature vector $V$)?
I divided the samples into four non-overlapping sets: $A$ ...
1
vote
0answers
45 views
Assessment of a single predictive model
I am facing a situation where I need to assess the predictive ability of a single (regression) model. I do not need to compare its performance with some other model, I just need to show that it ...
1
vote
0answers
140 views
Validation error less than training error — implications?
I am running a neural net to predict used car prices, sample size is 800. Using both 10-fold cross validation (10 times) and 1/3 holdback (10 times), the $R^2$ for training is about 0.60 and for ...
1
vote
0answers
82 views
Intraclass correlation to validate a measure
I would like to understand whether the output of an algorithm I developed agrees with the ratings of external observers. The number of observers is 10 and each of them rated on 2 3-point-Likert items ...
1
vote
1answer
291 views
How to select validation data when training a neural network?
I am training a neural network with time dependent financial data. In order to avoid overfitting I would like to stop the training at the point where my neural network stops improving on a set of ...
