Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

4
votes
3answers
245 views

Is overfitted model with higher AUC on test sample better than not overfitted one

i am participating in a challange in which I have created a model that performs 70% AUC on train set and 70% AUC on hold-out test set. The other participant has created a model that performs 96% AUC ...
1
vote
0answers
29 views

Gold Standard data for training but not validation

My goal is to determine which among the three classification algorithms perform better { Logist Reg or Neural Network or SVM }. I have a training dataset and the ...
2
votes
0answers
31 views

Choosing evaluation measure for non-parametric clustering

I have to cluster some data using non-parametric clustering technique which is given in this paper. After all the cluster evaluation measure used in this paper is Normalized Mutual Information as they ...
0
votes
0answers
4 views

Add Structure on Experimental semivariogram and validation of semivariogram [on hold]

im fairly new to this so please be patient with me. I am using surpac 6.3.2 and i want to know when to add a "structure" to the experimental semivar, also, once i have completed a semivar, how to i ...
0
votes
0answers
17 views

how to measure clustering task with unlabelled data set [duplicate]

I wanna know, how to measure the accuracy of a clustering method when we deal with data set without an a priori knowledge about class belonging ? (the data used for the clustering task, do not contain ...
-1
votes
0answers
21 views

cluster external validation [closed]

I am using ELKI in order to perform location clustering with DBSCAN and OPTICS. My data set include 30 participants but it is not labeled but I do have pair of coordinates (e.g. home, work, etc) as ...
4
votes
0answers
53 views

How to Validate a Monte Carlo Simulation

I have historical data of a production process, and I've being asked to build a simulation model to predict its performance in the future. Using the historical data, I've being able to obtain the ...
1
vote
2answers
53 views

Repeatedly split data in training (0.75) and test (0.25) for cross validation

What kind of cross validation is it called when we randomly split the data into 0.75 training and 0.25 test data set. And this split is done 1000 times.
1
vote
1answer
32 views

What is the proper name for “unknown data” set in machine learning?

As far as I know in practice the whole training set is usually split into training, validation and test[1] sets. Training set is used to train the model, validation to tune the parameters and test set ...
1
vote
0answers
9 views

What's the most appropriate way to derive and validate a model with hierarchical data

I am working on a model to predict the risk of some outcomes and could really use some advise: Let's say we have x number of patients, each patient have anywhere between 0 and y number of visits (...
0
votes
1answer
31 views

What are the alternatives to MMRE, PRED and MdMRE for validation?

I am working over the statistical validation of data. Till now I have computed MMRE, PRED and MdMRE. But I need alternatives to these because MRE is sensitive to data with large MRE's.
0
votes
0answers
26 views

What is the formula of Median Magnitude of Relative Error? (MdMRE)

I'm familiar with these terms 1 - MRE Mean Relative Error 2 - MMRE Mean Magnitude of Relative Error I need to compute MdMRE which is Median Mean Relative Error. I searched on the net but didn't ...
0
votes
0answers
13 views

Validating a qualitative method with quantitative data

I have developed an algorithm to detect micro events in sleep. These events have duration of a couple seconds, and in my data set each subject has around 100 of these for a full night being ~6-8 hours....
0
votes
0answers
10 views

How to split a survival data such that the proportion of events and censoring are equal in both groups

I need to develop a prognostic model, i have the survival data, and i need to split into validation set and training set. However, I want the Ratio of event to censoring in both sets to be equal. so ...
0
votes
1answer
17 views

How to address edge relations between training set and test set?

Suppose we are working on some sort of classification problem, and we have subdivided our data into a training set and a test set (or validation set, or etc.). We wish to prepare the data in the ...
0
votes
0answers
24 views

Why classification accuracy in validation set gets lower if validation cost also gets lower?

I'm training neural network for some simple classification task using tensorflow and have 2 output neurons, using softmax classification. My question is why accuracy on validation set gets lower when ...
0
votes
0answers
16 views

statistical tolerance

My request is: I have data (sums of money) and someone calculated tolerance limits for the data. I want to validate these tolerance limits with statistical methods/arguments. Does anyone know ...
4
votes
0answers
82 views

optimism-corrected regression coefficients using Frank Harrell's method?

I used a regularized (LASSO) cox regression to estimate relapse times of patients and used Frank Harrell's bootstrapping method to obtain an optimism-corrected performance estimate of my model. I am ...
0
votes
0answers
10 views

A new formula to estimate seminal vesicle volume? Which method to validate it?

I developed a new formula to estimate seminal vesicles volume, and tried it on 75 cases. I want to compare this new formula-method with golden standard method? I think i can use correlation(Pearson) ...
1
vote
0answers
42 views

How to validate a Poisson GLMM model?

I’m using the glmer function from the lme4 package in R to model species richness adjacent ...
0
votes
0answers
23 views

Self Organizing Map and input normalizing

I've been playing around with self organizing maps (SOM) recently. I tried to implement a simple example. You can see the training implementation function gist here and full contained SOM example ...
1
vote
0answers
23 views

ARIMA model over- or underfitting: compare training and validation performance

I'm doing research using seasonal and nonseasonal ARIMA models. Here's the result of model identification: Based on many sources, Your model is overfitting your training data when you see that ...
0
votes
0answers
21 views

Comparing a regression model with a single unit to the original model with all units

Background information: I have a regression model consisting of 230 companies (entity) for 20 years The model has 9 X-variables, and the P-value < 1% for the whole model The Y-...
1
vote
0answers
19 views

CNN training and overfitting

When testing the training of a CNN code with a small data set (approx 2560 images each for training and validation), what is over-fitting and how can it be mitigated? Arnold
0
votes
2answers
97 views

Model fitting: resampling the validation set to obtain distributions of test statistic

I see many descriptions of splitting the data set into a training part, a validation part and a test part. We train our models on the training part and choose the best model using the validation part, ...
1
vote
2answers
27 views

Validation set in presence of cross-validation

I am new to machine learning and want to ask regarding a confusion I have. I have a data set which is labeled and I want to do supervised learning. My question is related to cross-validation and ...
0
votes
0answers
21 views

why don't my CNN validation errors decrease?

I'm running theano_alexnet. I found that with greater numbers of iterations (20,000) the training cost and training error rates began to decrease (6.9 to 4.9 and 99% to 93%, respectively).When I ...
0
votes
0answers
31 views

R: Interpreting Fowlkes–Mallows index output for comparing dendrograms for hierarchical clustering

I have two data sets which contains information about subsystems in a bacterial metabolic model DataSet1: Behavior data of the subsystems DataSet2: Structural data of the same subsystems Then perform ...
0
votes
1answer
42 views

What should be validation strategy?

I am building CTR(https://en.wikipedia.org/wiki/Click-through_rate) Click prediction model with different (61) variables.Dependent variable is weather 0/1( click).I have build logistic regression ...
0
votes
2answers
103 views

Classification accuracy increasing while overfitting

I'm training a classification model, and these are the plots for accuracy and loss history. Besides the fact that the learning rate is too large, what I understand is that the model start ...
0
votes
1answer
25 views

validating model in machine learning - what does it mean in reality (intuition)

Could someone explain (in simple way) what does mean of validating model ? I tried to understand it, but I didn't managed to. I can do cross-validation, but I am not sure about if it is validation....
0
votes
0answers
11 views

How to summarize MSE across groups?

I have a large amount of time series data collected for different groups over a 30 year period (dataset x) that can be broken down by sub-group. and corresponding time series data (dataset y) from a ...
1
vote
0answers
20 views

When I can say a dataset is imbalanced?

I have a data set with only two outputs: positive and negative The ratio of positive:negative is 3.5:1 In this case, is my data set unbalanced? If so, what metric I should use to report the ...
1
vote
0answers
11 views

Does it make sense to use my in-sample data to look at goodness of fit for my model?

Basically, I came across an article where the authors first ran a logistic regression on a data set to predict the probability (q=demand) of buying their product, as a function of price p and various ...
0
votes
1answer
31 views

Topic Modeling Dataset for Code Verification

I am trying to write up a Gibbs sampling Latent Dirichlet Allocation function for myself in R, and wanted to run it on a dataset where the true classification of ...
1
vote
0answers
58 views

Cluster validation method for no cluster labels and differently sized clusters

I'm primarily a programmer and have little to no training in formal maths or statistics of any kind. I'm working on my dissertation (which foolishly is about clustering data), the process is ...
0
votes
0answers
37 views

Can I use the Xie-Beni index to validate data transformation parameters in fuzzy c-means clustering?

I am using fuzzy c-means algorithm to cluster my data in various feature spaces and the results differ depending on what kind of transformation I perform on my raw data. I want to know if using the ...
2
votes
1answer
58 views

Can we gain by merging validation and test set?

Reading this, Cross-validation including training, validation, and testing. Why do we need three subsets? I realized that if we can reduce the variance of the model performance, I wouldn't need the ...
0
votes
0answers
23 views

How to bootstrap validate regression model that involves removing outliers?

Suppose I have the following modelling process: Fit simple linear regression to whole data. Identify outliers, in the sense of having studentized residuals greater than a threshold, and remove them. ...
0
votes
0answers
17 views

External validation of a group's machine learning process

Say there's a group (that is opaque to you) that's heavily using machine learning methods to produce some outputs. You don't have access to their input data or code but can ask high level questions ...
0
votes
0answers
13 views

What sort of cross validation is this?

I've always tested my classification techniques using non-standardised trial and error but I'm interested to see which category my techniques fall under. They seem to fall under several but I'm not ...
0
votes
2answers
23 views

What do we learn from a test set?

Suppose I split my data into two parts -- a training set (having 80% of my data) and a testing (20%) set. I train a model on my training set, and test it on the test set. What do we learn from ...
1
vote
0answers
3 views

Validation of prediction interval for count data

I have developed a random-effects (frailty) survival model for repeated events which enables calculating individual-specific mean and prediction interval for the cumulative incidence (rate) of future ...
3
votes
1answer
44 views

Calculate Brier score and cindex for AFT model in R

While with Cox PH model, package pec in R provides several useful functions to validate the models based on cindex and brier score, with AFT model, pec does not support object with class survreg. Is ...
1
vote
1answer
107 views

cross validation and validation( to find RMS error and correlation ) in matlab

Dear Experts; i have text data (sample points are 324) of different climatic parameters. 3rd column of each text file was contained some missing or NaN data. Using Scatter data interpolation in ...
0
votes
0answers
15 views

How to find the accuracy of a multiple linear regression model against existing dataset ?`

I need to make a statement of accuracy of my predictive model against existing dataset , how should I proceed ahead with it?
1
vote
0answers
38 views

How to split training data for parameter tuning and accurate out-of-sample predictions

I am currently working on a kaggle competition; ignoring the details, each training & test set example, in this competition, essentially consists of product and search term pairs. Given each of ...
1
vote
0answers
50 views

Computing c-index for an external validation of a Cox PH model with Stata

I have developed a Cox prognostic model based and internally validated it using bootstrapping. I also evaluated its calibration using Harrell’s $c$ statistic and Somers' $D$. I have also checked ...
0
votes
0answers
28 views

Calibration test of cox-model in external validation using stata

I'm planning to validate a cox-risk prediction model in external dataset. As a pert of the validation process, I want to test the calibration of my model in the external dataset. I would appreciate if ...