Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

2
votes
3answers
749 views

Back propagation neural network data input advice

I am currently doing a project which involves pothole detection and neural networks. So far, I have an Android phone that reads Accelerometer readings and writes the X,Y,Z Axis aswell as the Amplitude ...
2
votes
4answers
492 views

Statistical tests to do data quality checks

I want to identify bad responses in the data we get from our vendors. To assess the quality of the data, what statistical tests can i apply to identify bad respondents? I performed Discriminant ...
23
votes
2answers
2k views

How to draw valid conclusions from “big data”?

"Big data" is everywhere in the media. Everybody says that "big data" is the big thing for 2012, e.g. KDNuggets poll on hot topics for 2012. However, I have deep concerns here. With big data, ...
2
votes
0answers
49 views

Validating previous analysis with new data

I am interested in trying to validate some results from a previous experiment. The experiment involves selecting probes of interest that separate our two populations of interest, e.g. smoker vs. ...
0
votes
1answer
263 views

What is a good academic citation for cross-validation?

I performed a holdout cross-validation analysis on a multilevel model fit. The purpose of this was to show that we didn't have a problem with over-fitting, for which it worked just fine. Now we are ...
5
votes
1answer
114 views

What do you do with your testing data?

Suppose you perform a parameter selection study in an environment with not-so-many data points. You divide the available data into training and testing set and you plan to validate the future model on ...
5
votes
1answer
244 views

Name of mean absolute error analogue to Brier score?

Yesterday's question Determine accuracy of model which estimates probability of event got me curious about probability scoring. The Brier score $$\frac{1}{N}\sum\limits _{i=1}^{N}(prediction_i - ...
3
votes
2answers
406 views

Logistic regression performs better on validation data

Recently I've been building a model using logistic regression. To my suprisise LIFT chart looks better on the validation data than on the training data, the same is with ROC. All variables in the ...
2
votes
0answers
94 views

Validation of data partitioning

I've some numerical sequence of data monitored over time $v(i)$, which belongs to one of two classes, 0 or 1. I monitor the variable several times, obtaining a sequence associated to 0 or 1. I use ...
30
votes
5answers
23k views

What is the difference between test set and validation set?

I found this confusing when I use the neural network toolbox in Matlab. It divided the raw data set into three parts: training set validation set test set I notice in many training or learning ...
4
votes
1answer
334 views

Compare modeled (fitted) paired data to actual data in forecasting problem (Excel sheet included)

In my endeavor to learn forecasting and improve my statistical knowledge, I've decided to forecast the population of a certain area. I've attached the excel file that I used. I know I'm using excel ...
1
vote
1answer
118 views

Difference between Step validation and Cross validation

I'm trying to model some chemical data from a series of spectra. I've gone through the preprocessing step and selected "autoscale" (I'm using Pirouette). And when it comes to validation I'm a little ...
1
vote
1answer
422 views

Why are most standard goodness of fit tests based only on continuous distributions?

I tried to search info regarding this fact but I don't really understand why most of the standard goodness of fit tests (e.g. Kolmogorov-Smirnov, Anderson-Darling, a part of the Chi-square test, ...
4
votes
1answer
1k views

Best practices for measuring and avoiding overfitting?

I am developing automated trading systems for the stock market. The big challenge has been overfitting. Can your recommend some resources describing methods for measuring and avoiding overfitting? I ...
1
vote
1answer
372 views

Validating a predicted standard deviation

I would like to validate the standard deviation which a model reports, because I think it is underestimated, because in the procedure some model selection is involved. After running the model on a ...
20
votes
4answers
501 views

As a reviewer, can I justify requesting data and code be made available even if the journal does not?

As science must be reproducible, by definition, there is increasing recognition that data and code are an essential component of the reproduciblity, as discussed by the Yale Roundtable for data and ...
2
votes
1answer
757 views

Weight variables for predictive model

I received a question today that I wasn't exactly sure how to answer. I have built a predictive model using a fairly basic logistic regression that works pretty well and fits our business needs. ...
1
vote
1answer
728 views

How to plot AIC values when using the leaps package?

Does anybody know how to plot all AIC values for different size models, when using the command regsubsets from the package ...
3
votes
1answer
645 views

How to choose training and test sets

I would like to propose a single model (decision tree), that is very variable, and validate it. I have choosen parameters after I had obtained good quality measures with a cross-validation. I could ...
4
votes
2answers
940 views

Variance explained of a mixed effects model in a new data set

I'm using the following function to calculate Edwards R^2 (formula 19 in Edwards et al. 2008) of a mixed effects model (I hope the implementation is correct): ...
2
votes
1answer
213 views

Validating a paper questionnaire in a web-based format

I would like to validate a commonly used questionnaire for internet-based administration. What statistical tests should I be using?
1
vote
0answers
163 views

Minimal number of samples/conversions for statistical validity

We are measuring conversion rates (% of visitors who bought) on an e-commerce site. The test apply to a segment of visitors who meet specific criteria (for example people from a certain country). ...
5
votes
2answers
5k views

How to make representative sample set from a large overall dataset?

What are the statistical techniques to create a sample set, which is representative of the entire population (with a known confidence level)? Also, How to validate, if the sample fits the overall ...
6
votes
3answers
2k views

What is a consistency check?

I was asked such a question as "Did you do any consistency check in your daily work?" during a phone interview for a Biostatistician position. I don't know what to answer. Any information is ...
3
votes
1answer
209 views

What resources/methods exist for testing/validation or evaluation of Statistical Methods

I'm interested in the process of testing or validating a particular implementation of a statistical method, and what datasets and/or published analysis exist that could be used to do this in practice. ...