Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

0
votes
2answers
158 views

highly sporadic validation error during training with multilayer perceptron

I'm encountering an issue where a classifier I'm developing reports validation errors during training that span a wide range of values without consistently decreasing over time. Unfortunately, I'm new ...
0
votes
1answer
10 views

Is an SLA pulled by just the calls answered by a certain group and not for the whole Queue valid?

I am trying to determine if the following is a valid SLA, I was presented with this as a fix for us not being able to differentiate the SLA between multiple groups answering calls: 1 Queue Two ...
1
vote
0answers
20 views

validating a gaussian process fitted to data

I am relatively new to applying Gaussian processes to data. I come from a math background but the most popular literature on it seems to be from a machine learning perspective and not from a ...
0
votes
0answers
12 views

undersampling-unbalanced data

what are the correct steps for undersampling in classifying models? for instance if a have an unbalanced dataset with 950 non event and 50 events I will undersample creating a dataset with a 50-50% ...
4
votes
1answer
216 views

Minimal number of samples/conversions for statistical validity

We are measuring conversion rates (% of visitors who bought) on an e-commerce site. The test apply to a segment of visitors who meet specific criteria (for example people from a certain country). ...
-1
votes
1answer
11 views

datav alidation on lexcel [closed]

I have given data validation for Std. warehouse as "YGCTCAP1" and "YGCTBAP1" for the first row i entered YGCTCAP1 by drop down, and i drag down the cell, it in became YGCTCAP2, --3,--4 ETC , ...
0
votes
0answers
17 views

Are multiple testing corrections necessary when validating against an already processed and published database?

Short version: Do I have to apply multiple testing corrections when analyzing a single variable (from a multidimensional dataset) I have previously found to be relevant and validated in the lab using ...
7
votes
3answers
416 views

How to select or validate the selection of a clustering method?

One of the biggest issue with cluster analysis is that we may happen to have to derive different conclusion when base on different clustering methods used (including different linkage methods in ...
0
votes
0answers
7 views

Validating a multinomial regression model

I have to validate a multinomial regression,i have a response made by 6 groups and 12 variables. My problem is that through Chisquare test my variables are not significant but my model through ...
0
votes
0answers
13 views

Is it the right to split a shuffled training set into training and validation sets?

I want to create a validation set for CIFAR10 dataset which can be found here. The training set has a file named train.txt which contains a list of image's path ...
1
vote
1answer
18 views

How to validate simulation result with real data

I have made a computer simulation, which is a model of a machine process step in a real factory. This factory keeps track of certain KPIs (key performance indicators), like utilization (% of time in ...
3
votes
0answers
45 views

Cross-validation scheme used in the Introduction to Statistical Learning, Chapter 6, Lab 3

I've been really enjoying the Introduction to Statistical Learning textbook so far, and I'm currently working my way through chapter 6. I realize that I am very confused by the process used in lab 3 ...
0
votes
0answers
16 views

Silhouette score significance: what is a significant increase in silhouette score?

I'm aware a silhouette score ranges from -1 to 1. But what can be considered a significant increase? 0.1 to 0.2 (because 100%) or 0.5 to 0.6? Obviously higher is better, but is there some measure of ...
2
votes
3answers
35 views

Comparing silhouette scores between different datasets (having different number of variables)

Full Question Experiment 1 clustered data using variables X and Y. Experiment 2 clustered data using variables X, Y and Z (i.e. a third variable was added). Would it be valid to compare silhouette ...
3
votes
1answer
265 views

Validation of a questionnaire in a new population

I have 400 responses to a 20 item questionnaire which purports to measure an attitudinal constuct in medical students. The instrument was validated in the US for a single year of medical students and ...
4
votes
4answers
134 views

Should I get 100% classification accuracy on training data?

I've been getting inconsistent results with a binary classification problem I'm trying to solve using a linear classifier and a custom feature extraction pipeline, and decided to do a quick check of ...
1
vote
1answer
30 views

Comparing performance of two k-fold cross-validated models

I'm developing two k-fold cross-validated models, based on two different data sets, but using the same variables. I plan to then apply both models to each data set and calculate a few model ...
7
votes
1answer
1k views

Within-group sum of squares of cluster

I have a multivariate dataset for which I have only a table including the cross-wise Euclidean distances between all points and a list giving the assignment of each point to one of several clusters. ...
1
vote
1answer
19 views

How to prove two methods are “similar”?

We are testing two imaging machines. One of the machines has been validated numerous times. However, we built the new machine and we need to prove that it is similar/comparable to the validated ...
1
vote
3answers
62 views

Repeatedly split data in training (0.75) and test (0.25) for cross validation

What kind of cross validation is it called when we randomly split the data into 0.75 training and 0.25 test data set. And this split is done 1000 times.
1
vote
1answer
38 views

Gold Standard data for training but not validation

My goal is to determine which among the three classification algorithms perform better { Logist Reg or Neural Network or SVM }. I have a training dataset and the ...
5
votes
3answers
300 views

Is overfitted model with higher AUC on test sample better than not overfitted one

i am participating in a challange in which I have created a model that performs 70% AUC on train set and 70% AUC on hold-out test set. The other participant has created a model that performs 96% AUC ...
2
votes
0answers
35 views

Choosing evaluation measure for non-parametric clustering

I have to cluster some data using non-parametric clustering technique which is given in this paper. After all the cluster evaluation measure used in this paper is Normalized Mutual Information as they ...
0
votes
0answers
17 views

how to measure clustering task with unlabelled data set [duplicate]

I wanna know, how to measure the accuracy of a clustering method when we deal with data set without an a priori knowledge about class belonging ? (the data used for the clustering task, do not contain ...
4
votes
0answers
59 views

How to Validate a Monte Carlo Simulation

I have historical data of a production process, and I've being asked to build a simulation model to predict its performance in the future. Using the historical data, I've being able to obtain the ...
5
votes
1answer
500 views

Prediction evaluation metric for panel/longitudinal data

I would like to evaluate several different models that provide predictions of behavior at a monthly level. The data is balanced, and $n=$100,000 and $T=$12. The outcome is attending a concert in a ...
1
vote
1answer
34 views

What is the proper name for “unknown data” set in machine learning?

As far as I know in practice the whole training set is usually split into training, validation and test[1] sets. Training set is used to train the model, validation to tune the parameters and test set ...
4
votes
0answers
86 views

optimism-corrected regression coefficients using Frank Harrell's method?

I used a regularized (LASSO) cox regression to estimate relapse times of patients and used Frank Harrell's bootstrapping method to obtain an optimism-corrected performance estimate of my model. I am ...
2
votes
2answers
2k views

How to plot AIC values when using the leaps package?

Does anybody know how to plot all AIC values for different size models, when using the command regsubsets from the package leaps?...
4
votes
2answers
628 views

Logistic regression performs better on validation data

Recently I've been building a model using logistic regression. To my suprisise LIFT chart looks better on the validation data than on the training data, the same is with ROC. All variables in the ...
1
vote
0answers
12 views

What's the most appropriate way to derive and validate a model with hierarchical data

I am working on a model to predict the risk of some outcomes and could really use some advise: Let's say we have x number of patients, each patient have anywhere between 0 and y number of visits (...
0
votes
1answer
33 views

What are the alternatives to MMRE, PRED and MdMRE for validation?

I am working over the statistical validation of data. Till now I have computed MMRE, PRED and MdMRE. But I need alternatives to these because MRE is sensitive to data with large MRE's.
0
votes
0answers
37 views

What is the formula of Median Magnitude of Relative Error? (MdMRE)

I'm familiar with these terms 1 - MRE Mean Relative Error 2 - MMRE Mean Magnitude of Relative Error I need to compute MdMRE which is Median Mean Relative Error. I searched on the net but didn't ...
4
votes
2answers
2k views

Using Adaboost for feature selection?

Is it okay to use Adaboost to do feature selection (selecting a subset of dimensions $S$ from a high-dimensional feature vector $V$)? I divided the samples into four non-overlapping sets: $A$ (...
0
votes
0answers
13 views

Validating a qualitative method with quantitative data

I have developed an algorithm to detect micro events in sleep. These events have duration of a couple seconds, and in my data set each subject has around 100 of these for a full night being ~6-8 hours....
0
votes
0answers
10 views

How to split a survival data such that the proportion of events and censoring are equal in both groups

I need to develop a prognostic model, i have the survival data, and i need to split into validation set and training set. However, I want the Ratio of event to censoring in both sets to be equal. so ...
0
votes
1answer
17 views

How to address edge relations between training set and test set?

Suppose we are working on some sort of classification problem, and we have subdivided our data into a training set and a test set (or validation set, or etc.). We wish to prepare the data in the ...
0
votes
0answers
31 views

Why classification accuracy in validation set gets lower if validation cost also gets lower?

I'm training neural network for some simple classification task using tensorflow and have 2 output neurons, using softmax classification. My question is why accuracy on validation set gets lower when ...
0
votes
0answers
16 views

statistical tolerance

My request is: I have data (sums of money) and someone calculated tolerance limits for the data. I want to validate these tolerance limits with statistical methods/arguments. Does anyone know ...
2
votes
1answer
106 views

The effect of oversampling on the positive predictive value

I need to calculate the positive predictive value for a validation set for a rare event. The problem is that the validation set was oversampled for the rare event. The event occurs in 5 percent of the ...
1
vote
0answers
25 views

ARIMA model over- or underfitting: compare training and validation performance

I'm doing research using seasonal and nonseasonal ARIMA models. Here's the result of model identification: Based on many sources, Your model is overfitting your training data when you see that ...
0
votes
0answers
11 views

A new formula to estimate seminal vesicle volume? Which method to validate it?

I developed a new formula to estimate seminal vesicles volume, and tried it on 75 cases. I want to compare this new formula-method with golden standard method? I think i can use correlation(Pearson) ...
1
vote
0answers
56 views

How to validate a Poisson GLMM model?

I’m using the glmer function from the lme4 package in R to model species richness adjacent ...
0
votes
0answers
39 views

Self Organizing Map and input normalizing

I've been playing around with self organizing maps (SOM) recently. I tried to implement a simple example. You can see the training implementation function gist here and full contained SOM example ...
0
votes
0answers
21 views

Comparing a regression model with a single unit to the original model with all units

Background information: I have a regression model consisting of 230 companies (entity) for 20 years The model has 9 X-variables, and the P-value < 1% for the whole model The Y-...
0
votes
2answers
98 views

Model fitting: resampling the validation set to obtain distributions of test statistic

I see many descriptions of splitting the data set into a training part, a validation part and a test part. We train our models on the training part and choose the best model using the validation part, ...
1
vote
0answers
30 views

CNN training and overfitting

When testing the training of a CNN code with a small data set (approx 2560 images each for training and validation), what is over-fitting and how can it be mitigated? Arnold
1
vote
2answers
28 views

Validation set in presence of cross-validation

I am new to machine learning and want to ask regarding a confusion I have. I have a data set which is labeled and I want to do supervised learning. My question is related to cross-validation and ...
0
votes
0answers
28 views

why don't my CNN validation errors decrease?

I'm running theano_alexnet. I found that with greater numbers of iterations (20,000) the training cost and training error rates began to decrease (6.9 to 4.9 and 99% to 93%, respectively).When I ...
0
votes
0answers
36 views

R: Interpreting Fowlkes–Mallows index output for comparing dendrograms for hierarchical clustering

I have two data sets which contains information about subsystems in a bacterial metabolic model DataSet1: Behavior data of the subsystems DataSet2: Structural data of the same subsystems Then perform ...