Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

0
votes
0answers
8 views

why don't my CNN validation errors decrease?

I'm running theano_alexnet. I found that with greater numbers of iterations (20,000) the training cost and training error rates began to decrease (6.9 to 4.9 and 99% to 93%, respectively).When I ...
0
votes
0answers
18 views

R: Interpreting Fowlkes–Mallows index output for comparing dendrograms for hierarchical clustering

I have two data sets which contains information about subsystems in a bacterial metabolic model DataSet1: Behavior data of the subsystems DataSet2: Structural data of the same subsystems Then perform ...
0
votes
1answer
39 views

What should be validation strategy?

I am building CTR(https://en.wikipedia.org/wiki/Click-through_rate) Click prediction model with different (61) variables.Dependent variable is weather 0/1( click).I have build logistic regression ...
0
votes
2answers
39 views

Classification accuracy increasing while overfitting

I'm training a classification model, and these are the plots for accuracy and loss history. Besides the fact that the learning rate is too large, what I understand is that the model start ...
0
votes
1answer
24 views

validating model in machine learning - what does it mean in reality (intuition)

Could someone explain (in simple way) what does mean of validating model ? I tried to understand it, but I didn't managed to. I can do cross-validation, but I am not sure about if it is ...
0
votes
0answers
7 views

How to summarize MSE across groups?

I have a large amount of time series data collected for different groups over a 30 year period (dataset x) that can be broken down by sub-group. and corresponding time series data (dataset y) from a ...
1
vote
0answers
20 views

When I can say a dataset is imbalanced?

I have a data set with only two outputs: positive and negative The ratio of positive:negative is 3.5:1 In this case, is my data set unbalanced? If so, what metric I should use to report the ...
1
vote
0answers
11 views

Does it make sense to use my in-sample data to look at goodness of fit for my model?

Basically, I came across an article where the authors first ran a logistic regression on a data set to predict the probability (q=demand) of buying their product, as a function of price p and various ...
0
votes
1answer
24 views

Topic Modeling Dataset for Code Verification

I am trying to write up a Gibbs sampling Latent Dirichlet Allocation function for myself in R, and wanted to run it on a dataset where the true classification of ...
1
vote
0answers
50 views

Cluster validation method for no cluster labels and differently sized clusters

I'm primarily a programmer and have little to no training in formal maths or statistics of any kind. I'm working on my dissertation (which foolishly is about clustering data), the process is ...
0
votes
0answers
30 views

Can I use the Xie-Beni index to validate data transformation parameters in fuzzy c-means clustering?

I am using fuzzy c-means algorithm to cluster my data in various feature spaces and the results differ depending on what kind of transformation I perform on my raw data. I want to know if using the ...
2
votes
1answer
54 views

Can we gain by merging validation and test set?

Reading this, Cross-validation including training, validation, and testing. Why do we need three subsets? I realized that if we can reduce the variance of the model performance, I wouldn't need the ...
0
votes
0answers
20 views

How to bootstrap validate regression model that involves removing outliers?

Suppose I have the following modelling process: Fit simple linear regression to whole data. Identify outliers, in the sense of having studentized residuals greater than a threshold, and remove them. ...
0
votes
0answers
17 views

External validation of a group's machine learning process

Say there's a group (that is opaque to you) that's heavily using machine learning methods to produce some outputs. You don't have access to their input data or code but can ask high level questions ...
0
votes
0answers
13 views

What sort of cross validation is this?

I've always tested my classification techniques using non-standardised trial and error but I'm interested to see which category my techniques fall under. They seem to fall under several but I'm not ...
0
votes
2answers
17 views

What do we learn from a test set?

Suppose I split my data into two parts -- a training set (having 80% of my data) and a testing (20%) set. I train a model on my training set, and test it on the test set. What do we learn from ...
1
vote
0answers
3 views

Validation of prediction interval for count data

I have developed a random-effects (frailty) survival model for repeated events which enables calculating individual-specific mean and prediction interval for the cumulative incidence (rate) of future ...
3
votes
1answer
35 views

Calculate Brier score and cindex for AFT model in R

While with Cox PH model, package pec in R provides several useful functions to validate the models based on cindex and brier score, with AFT model, pec does not support object with class survreg. Is ...
1
vote
1answer
92 views

cross validation and validation( to find RMS error and correlation ) in matlab

Dear Experts; i have text data (sample points are 324) of different climatic parameters. 3rd column of each text file was contained some missing or NaN data. Using Scatter data interpolation in ...
0
votes
0answers
12 views

How to find the accuracy of a multiple linear regression model against existing dataset ?`

I need to make a statement of accuracy of my predictive model against existing dataset , how should I proceed ahead with it?
1
vote
0answers
35 views

How to split training data for parameter tuning and accurate out-of-sample predictions

I am currently working on a kaggle competition; ignoring the details, each training & test set example, in this competition, essentially consists of product and search term pairs. Given each of ...
1
vote
0answers
35 views

Computing c-index for an external validation of a Cox PH model with Stata

I have developed a Cox prognostic model based and internally validated it using bootstrapping. I also evaluated its calibration using Harrell’s $c$ statistic and Somers' $D$. I have also checked ...
0
votes
0answers
19 views

Calibration test of cox-model in external validation using stata

I'm planning to validate a cox-risk prediction model in external dataset. As a pert of the validation process, I want to test the calibration of my model in the external dataset. I would appreciate if ...
3
votes
1answer
36 views

Random Sample Validation

This is a real life scenario where I have just assumed responsibility for a new team The results of my team who perform legal contract abstractions are audited each month by our client using random ...
0
votes
0answers
19 views

Reposted: Calibration and Discrimination of Cox prognostic model with Stata [duplicate]

I'm planning to do an external validation of a Cox-prognostic model against another study and a published risk prediction equation. Yet, my previous exposure is limited validation of logistic model. ...
0
votes
0answers
20 views

nltk: odd outputs from bleu_score

For machine translation purposes I use bleu score, which seems to be the validation mechanism of choice (used in the sutskever 2014 sequence-to-sequence). The purpose is to get as high bleu as ...
1
vote
0answers
27 views

Should difference between accuracy of model on training data and testing data be considered for model selection?

Suppose I have two models (Model 1 and Model 2), Where Accuracy of Model 1 on test data is higher than that in Model 2 Difference between accuracy of model on test data and training data is higher ...
0
votes
1answer
37 views

Choosing number of samples to train a model

(On behalf of a colleague) I have performed some modelling based on a naïve Bayes classifiers model (weighted genomic risk score) and obtained reasonable ROCAUC results (used ROCR, pROC, and SDMtools ...
1
vote
2answers
53 views

Which balance strategy to learn from my very imbalanced dataset?

I'm using a deep learning approach on a dataset made of ~20 millions of elements, where each element has a TRUE or FALSE label. This dataset unfortunately is veeery imbalanced: I've 98% of falses and ...
6
votes
1answer
88 views

How to account for case weights when generating folds for K-fold CV?

I am currently working on a binary classification problem where each point in the dataset is paired with a case weight. Specifically, each record $(w_i, x_i, y_i)$ represents a distinct type of ...
0
votes
0answers
9 views

dataset construction

I am trying to build a dataset for benchmarking but for which the sizes of each group is quite different. I am feeling there is something a bit dodgy in my current methodology but cannot point it out. ...
5
votes
1answer
78 views

What can be inferred from this residual plot?

This is with reference to http://analyticspro.org/2016/03/05/r-tutorial-residual-analysis-for-regression/. For Residual plot (b), where the residuals are increasing linearly (with respect to the ...
0
votes
0answers
16 views

How to determine sample size for validation?

In my study, I have a prediction model built using some previous samples that I have. Now I want to validate this model in an independent data set. Basically, I will use my model to make predictions ...
0
votes
1answer
24 views

Model prediction validation – when to retrain?

After a trained/tested model has been put in production, what's the best approach to keep track of the validity of the model such that one knows when to retrain?
1
vote
1answer
18 views

How to interpret Q quality index from val.prob and compare different scoring rule results

I'm interested in learning more about the Q value produced by val.prob in the R package RMS and how it compares with the Brier score. I understand Q, the quality ...
0
votes
1answer
28 views

How to choose the relative sizes of training and validation sets?

When I work with the methods of data mining, the data is split in training and validations data samples (and sometimes test). I know training + validation = 100%. Which criteria can I use to find a ...
0
votes
1answer
79 views

Cross-validation techniques for time series data

What is an appropriate cross-validation technique for time series data? I have a daily 4 years time series data and fitting a SVM model by MATLAB R2015b: ...
0
votes
0answers
17 views

What would be simple way of calculating the area of a 3D PieChart's slices?

I have created a 3D Pie Chart able to be rotated. -> http://plnkr.co/edit/QIYu8sJUWPmxcby1ky9l?p=preview I did it to demonstrate how the visual perception of data in a Pie Chart can be distorted ...
0
votes
0answers
9 views

Method validation in non-regression problems

Assume I have a sample of r species/classes that follow a multinomial distribution, where the vast majority of classes only are observed once. Further, let us say I have k models that all are giving ...
3
votes
1answer
75 views

What is the difference between sensitivity analysis and model validation?

I read both wikipedia pages of sensitivity analysis and model validation (here, only linear regression validation) but I don't manage to find a way to separate these two terms. I have the impression ...
0
votes
0answers
18 views

Real-time accounting input valdiation

I'm currently fishing in troubled waters, as I've got a huge scope. I'm sorry in case this question had been asked already in before, however I couldn't find even a topic to look for with my small ...
0
votes
0answers
20 views

Sensitivity analysis for data validation

I have a dataset, which I believe contains errors. Basically my dataset contains recorded time intervals (in seconds) of visits to a certain place, that due to sensor oversensitivy have been ...
0
votes
0answers
26 views

What kind of bias am I introducing when I include Validation set (kept for model selection) in Train data?

As usual, I have three sets of data: Train, Validation and Test. So I use train data for model selection, where I select the model which would perform best on validation data. After selecting the best ...
0
votes
1answer
18 views

Evaluating a clustering against multilabels

I have a clustering of text documents, where each document is uniquely assigned to a cluster. I have a set of labels (keywords) attached to each document. That is, each label may be applied to many ...
0
votes
0answers
29 views

When is preferred the relative and stability-based cluster validation?

I need to validate a clustering algorithm result. I know that Cluster Validation is commonly divided into four categories: internal, external, relative and Stability-based criteria, where internal and ...
0
votes
0answers
36 views

Standard errors and confidence interval in cox regression model validation using RMS package

I am using RMS package of R to validate cox regression model with bootstrap. Please see the sample code below. I have three questions: (1) How do I request the standard errors and/or confidence ...
3
votes
1answer
62 views

Where can I find tests that validate the output of popular statistical software? (e.g. R, SPSS, SAS)

This has been difficult to search on Google. I've looked on R's website as well. I assume there are test suites that validate that R, SPSS, SAS, et al are giving the right output for a given analysis? ...
0
votes
1answer
33 views

Can I validate results after statistical hypothesis testing?

I have data from one clinical test for 450 patients. Depending upon my objective, I divided data into 7 different categories to compare two groups, in each category (inherited from parents and not ...
0
votes
0answers
36 views

Comparing two distributions (simulation vs. measurement) for describing model performance

On the one hand we use a complex mathematical model to determine the result of a process. The model will iterate e.g. 2000 times, each time using random samples for some unknown input parameters ...
0
votes
0answers
5 views

sampling size for binary group estimation (under randomized validation)?

Having binary groups with n members (size of several thousand or more members which can have 0 to ...