Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

1
vote
0answers
34 views

Should difference between accuracy of model on training data and testing data be considered for model selection?

Suppose I have two models (Model 1 and Model 2), Where Accuracy of Model 1 on test data is higher than that in Model 2 Difference between accuracy of model on test data and training data is higher ...
0
votes
1answer
41 views

Choosing number of samples to train a model

(On behalf of a colleague) I have performed some modelling based on a naïve Bayes classifiers model (weighted genomic risk score) and obtained reasonable ROCAUC results (used ROCR, pROC, and SDMtools ...
1
vote
2answers
60 views

Which balance strategy to learn from my very imbalanced dataset?

I'm using a deep learning approach on a dataset made of ~20 millions of elements, where each element has a TRUE or FALSE label. This dataset unfortunately is veeery imbalanced: I've 98% of falses and ...
6
votes
1answer
92 views

How to account for case weights when generating folds for K-fold CV?

I am currently working on a binary classification problem where each point in the dataset is paired with a case weight. Specifically, each record $(w_i, x_i, y_i)$ represents a distinct type of person,...
0
votes
0answers
11 views

dataset construction

I am trying to build a dataset for benchmarking but for which the sizes of each group is quite different. I am feeling there is something a bit dodgy in my current methodology but cannot point it out. ...
5
votes
1answer
91 views

What can be inferred from this residual plot?

This is with reference to http://analyticspro.org/2016/03/05/r-tutorial-residual-analysis-for-regression/. For Residual plot (b), where the residuals are increasing linearly (with respect to the ...
0
votes
0answers
20 views

How to determine sample size for validation?

In my study, I have a prediction model built using some previous samples that I have. Now I want to validate this model in an independent data set. Basically, I will use my model to make predictions ...
0
votes
1answer
25 views

Model prediction validation – when to retrain?

After a trained/tested model has been put in production, what's the best approach to keep track of the validity of the model such that one knows when to retrain?
1
vote
1answer
19 views

How to interpret Q quality index from val.prob and compare different scoring rule results

I'm interested in learning more about the Q value produced by val.prob in the R package RMS and how it compares with the Brier score. I understand Q, the quality ...
0
votes
1answer
30 views

How to choose the relative sizes of training and validation sets?

When I work with the methods of data mining, the data is split in training and validations data samples (and sometimes test). I know training + validation = 100%. Which criteria can I use to find a ...
0
votes
1answer
114 views

Cross-validation techniques for time series data

What is an appropriate cross-validation technique for time series data? I have a daily 4 years time series data and fitting a SVM model by MATLAB R2015b: ...
0
votes
0answers
17 views

What would be simple way of calculating the area of a 3D PieChart's slices?

I have created a 3D Pie Chart able to be rotated. -> http://plnkr.co/edit/QIYu8sJUWPmxcby1ky9l?p=preview I did it to demonstrate how the visual perception of data in a Pie Chart can be distorted ...
0
votes
0answers
9 views

Method validation in non-regression problems

Assume I have a sample of r species/classes that follow a multinomial distribution, where the vast majority of classes only are observed once. Further, let us say I have k models that all are giving ...
3
votes
1answer
108 views

What is the difference between sensitivity analysis and model validation?

I read both wikipedia pages of sensitivity analysis and model validation (here, only linear regression validation) but I don't manage to find a way to separate these two terms. I have the impression ...
0
votes
0answers
20 views

Real-time accounting input valdiation

I'm currently fishing in troubled waters, as I've got a huge scope. I'm sorry in case this question had been asked already in before, however I couldn't find even a topic to look for with my small ...
0
votes
0answers
23 views

Sensitivity analysis for data validation

I have a dataset, which I believe contains errors. Basically my dataset contains recorded time intervals (in seconds) of visits to a certain place, that due to sensor oversensitivy have been ...
0
votes
0answers
27 views

What kind of bias am I introducing when I include Validation set (kept for model selection) in Train data?

As usual, I have three sets of data: Train, Validation and Test. So I use train data for model selection, where I select the model which would perform best on validation data. After selecting the best ...
0
votes
1answer
25 views

Evaluating a clustering against multilabels

I have a clustering of text documents, where each document is uniquely assigned to a cluster. I have a set of labels (keywords) attached to each document. That is, each label may be applied to many ...
0
votes
0answers
31 views

When is preferred the relative and stability-based cluster validation?

I need to validate a clustering algorithm result. I know that Cluster Validation is commonly divided into four categories: internal, external, relative and Stability-based criteria, where internal and ...
0
votes
0answers
49 views

Standard errors and confidence interval in cox regression model validation using RMS package

I am using RMS package of R to validate cox regression model with bootstrap. Please see the sample code below. I have three questions: (1) How do I request the standard errors and/or confidence ...
3
votes
1answer
62 views

Where can I find tests that validate the output of popular statistical software? (e.g. R, SPSS, SAS)

This has been difficult to search on Google. I've looked on R's website as well. I assume there are test suites that validate that R, SPSS, SAS, et al are giving the right output for a given analysis? ...
0
votes
1answer
34 views

Can I validate results after statistical hypothesis testing?

I have data from one clinical test for 450 patients. Depending upon my objective, I divided data into 7 different categories to compare two groups, in each category (inherited from parents and not ...
0
votes
0answers
38 views

Comparing two distributions (simulation vs. measurement) for describing model performance

On the one hand we use a complex mathematical model to determine the result of a process. The model will iterate e.g. 2000 times, each time using random samples for some unknown input parameters (...
0
votes
0answers
5 views

sampling size for binary group estimation (under randomized validation)?

Having binary groups with n members (size of several thousand or more members which can have 0 to ...
8
votes
2answers
359 views

Should final (production ready) model be trained on complete data or just on training set?

Suppose I trained several models on training set, choose best one using cross validation set and measured performance on test set. So now I have one final best model. Should I retrain it on my all ...
1
vote
0answers
19 views

What does this evaluation function mean?

I am solving a prediction problem where I am supposed to predict the number of clicks, given some historical Data. I am told that my predictions will be evaluated by a normalized weighted mean square ...
0
votes
0answers
21 views

What test can I use for Model Validation?

I ran the BTYD package in R which predicts the number of transactions that a customer is expected to make in the future.These expected values that I get are not integers but are in the form 0.14, 0.79,...
0
votes
0answers
17 views

Inproper model validation

Let us say that I have a data set D that contians N objects from K different species, words for example, where most of them occur one time. Now my advisor want's me to draw random samples with size ...
0
votes
0answers
53 views

What are the reasons why a classifier could produce bad results?

I know of four possible reasons: overfitting underfitting input data doesn't represent the problem (which I guess is underfitting) classifier isn't suitable (e.g. problem is not linear) Are there ...
4
votes
1answer
148 views

ML / train-test-validate: What is allowed when?

As someone getting started in machine learning, I am trying to get my head around the rules / good practices to follow when building, testing and validating supervised ML models in order not to ...
0
votes
1answer
83 views

How to pick the best model with cross validation?

Based on my understanding the leave one out cross validation is to hold a sample out as the test set and fit a model with remaining data and then calculate the error of prediction of the test sample ...
1
vote
1answer
57 views

What test shall I use to validate the use of a certain score to predict my outcome in a survival analysis?

I validate usage of a clinical cardiovascular score to predict the risk of dementia using data from a longitudinal study. Therefore, my outcome is binary (dementia yes or not) and the independent ...
0
votes
0answers
52 views

Which statistical test is preferred when comparing two predictive models?

Say, I have two predictive models A and B and recorded AUC (Area under curve) by running both the models on 20 different data sets. Now, if I want to find the best model out of these two for these ...
0
votes
0answers
27 views

Large condition number, but no effect on validation

My question is in regards to the relation between the condition number (or other multicollinearity diagnostics) and validation of a linear regression (lm in R). Specifically, I have extremely high ...
0
votes
0answers
6 views

How to model the behavior of Webservice and doing validation?

The goal is to find a smart way to validate the behavior of webservice such as CMS (wordpress, joomla). To give more detail, we want to be able based on observing host metrics ( cpu utilization,... ) ...
1
vote
0answers
23 views

Bootstrap within bootstrap?

I want to use .632 bootstrapping to validate an algorithm. The problem is that the algorithm itself also has bootstrapping procedures. Will that cause bias? Since bootstrapping with replacement will ...
2
votes
0answers
50 views

What should I do when logistic regression model doesn't perform well on test sample?

For logistic model, I divided dataset into two parts training sample (70% of 360 data point (observation)) and test sample (rest 30% of 360) randomly. After that I built logistic model on training ...
0
votes
1answer
60 views

Measures for Clustering Validations [duplicate]

I have unlabeled dataset and I'm using the hierarchical clustering to generate a groups from this data. I had a look to the lit and I found that there are two approaches to evaluate the clustering ...
1
vote
1answer
26 views

which paper(s) to cite when stating that out-of-sample validation is better than in-sample val

Validating a model (meaning, computing metrics on the prediction) using the training samples is, well, rubbish. Out-of sample validation should be preferred. This is well known by statisticians and ...
1
vote
0answers
16 views

The burden for verification/validation data

I seek a rigorous or otherwise proof for the need of verification/validation data. Typically it is suggested, after building a model, to do any validation of the model's health on a set of data that ...
0
votes
0answers
62 views

Validation Questionnaire Sample or format

I want to know on how to make a validation questionnaire for my research paper but I have no idea on how to create one. I searched about validation questionnaire but there's no format or example - ...
2
votes
2answers
80 views

The role of validation in estimation and hypothesis testing

Validation, with or without statistical/machine learning procedures, is used often, if not universally, in prediction. In estimation or hypothesis testing that does not seem to be the case, yet I don'...
0
votes
0answers
69 views

Calibration curve confidence intervals interpretation. Is to reduce a high Bias Corrected possible in Bootstrapping?

I have been working on a mixed generalized model, glmer, and now I am trying to validate it by a calibrate curve, and also would like generate its CIs. For this, I thought in Bootstraping and found ...
0
votes
0answers
21 views

Using one model to validate another? (Validation with no ground truth?)

My question is on model validation when there is no ground truth. Suppose I have a data set of attributes: $x = \{x_1,\ldots, x_n\}$ These attributes are different attributes that describe animals ...
2
votes
2answers
84 views

Double (nested,wrapper) CrossValidation - final trained model

I'm performing a study where I'm selecting kernel type and hyperparameters in an inner CV loop and an outer loop doing 10-fold CV (using SVR). The output is 10 trained models and performance measures. ...
0
votes
0answers
34 views

Avoiding over/under fitting & validation set size

I'm using a PLS model associated to FT-NIR spectroscopy to predict some properties of biodiesel. At the moment, I'm trying to optimize the number of factors I should use to do the calculations. Here,...
0
votes
0answers
118 views

SOM Validation method

I made a project using Self Organizing Map (SOM) for clustering multi dimensions data in R. But I have difficulty to find what method that I can use to: validate the clustering result based on data ...
0
votes
0answers
18 views

How to account for drift in system identification output?

I am trying to identify the lateral dynamics of a vehicle. I use steering angle as input, and lateral displacement, lateral velocity and yaw rate as output. I use idss method of MATLAB. The issue ...
1
vote
1answer
56 views

What are the assumptions to test on unseen data to use a predictive model?

I am trying to understand what are the assumptions I have to test, to use a predictive model on unseen data. After the validation phase of my predictive model and the estimation of its accuracy on a ...
0
votes
0answers
9 views

Correcting for multiple data points

I am validating a new measure of therapist competency in psychology. I am getting 4 clients who have taken part in 40 sessions (10 each) to answer a questionnaires about their therapy sessions (...