Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

1
vote
0answers
18 views

validating a gaussian process fitted to data

I am relatively new to applying Gaussian processes to data. I come from a math background but the most popular literature on it seems to be from a machine learning perspective and not from a ...
0
votes
0answers
11 views

undersampling-unbalanced data

what are the correct steps for undersampling in classifying models? for instance if a have an unbalanced dataset with 950 non event and 50 events I will undersample creating a dataset with a 50-50% ...
-1
votes
1answer
11 views

datav alidation on lexcel [on hold]

I have given data validation for Std. warehouse as "YGCTCAP1" and "YGCTBAP1" for the first row i entered YGCTCAP1 by drop down, and i drag down the cell, it in became YGCTCAP2, --3,--4 ETC , ...
0
votes
0answers
16 views

Are multiple testing corrections necessary when validating against an already processed and published database?

Short version: Do I have to apply multiple testing corrections when analyzing a single variable (from a multidimensional dataset) I have previously found to be relevant and validated in the lab using ...
0
votes
0answers
7 views

Validating a multinomial regression model

I have to validate a multinomial regression,i have a response made by 6 groups and 12 variables. My problem is that through Chisquare test my variables are not significant but my model through ...
0
votes
0answers
13 views

Is it the right to split a shuffled training set into training and validation sets?

I want to create a validation set for CIFAR10 dataset which can be found here. The training set has a file named train.txt which contains a list of image's path ...
1
vote
1answer
18 views

How to validate simulation result with real data

I have made a computer simulation, which is a model of a machine process step in a real factory. This factory keeps track of certain KPIs (key performance indicators), like utilization (% of time in ...
3
votes
0answers
44 views

Cross-validation scheme used in the Introduction to Statistical Learning, Chapter 6, Lab 3

I've been really enjoying the Introduction to Statistical Learning textbook so far, and I'm currently working my way through chapter 6. I realize that I am very confused by the process used in lab 3 ...
0
votes
0answers
16 views

Silhouette score significance: what is a significant increase in silhouette score?

I'm aware a silhouette score ranges from -1 to 1. But what can be considered a significant increase? 0.1 to 0.2 (because 100%) or 0.5 to 0.6? Obviously higher is better, but is there some measure of ...
4
votes
4answers
132 views

Should I get 100% classification accuracy on training data?

I've been getting inconsistent results with a binary classification problem I'm trying to solve using a linear classifier and a custom feature extraction pipeline, and decided to do a quick check of ...
2
votes
3answers
34 views

Comparing silhouette scores between different datasets (having different number of variables)

Full Question Experiment 1 clustered data using variables X and Y. Experiment 2 clustered data using variables X, Y and Z (i.e. a third variable was added). Would it be valid to compare silhouette ...
1
vote
1answer
30 views

Comparing performance of two k-fold cross-validated models

I'm developing two k-fold cross-validated models, based on two different data sets, but using the same variables. I plan to then apply both models to each data set and calculate a few model ...
1
vote
1answer
18 views

How to prove two methods are “similar”?

We are testing two imaging machines. One of the machines has been validated numerous times. However, we built the new machine and we need to prove that it is similar/comparable to the validated ...
5
votes
3answers
297 views

Is overfitted model with higher AUC on test sample better than not overfitted one

i am participating in a challange in which I have created a model that performs 70% AUC on train set and 70% AUC on hold-out test set. The other participant has created a model that performs 96% AUC ...
1
vote
1answer
38 views

Gold Standard data for training but not validation

My goal is to determine which among the three classification algorithms perform better { Logist Reg or Neural Network or SVM }. I have a training dataset and the ...
2
votes
0answers
35 views

Choosing evaluation measure for non-parametric clustering

I have to cluster some data using non-parametric clustering technique which is given in this paper. After all the cluster evaluation measure used in this paper is Normalized Mutual Information as they ...
0
votes
0answers
17 views

how to measure clustering task with unlabelled data set [duplicate]

I wanna know, how to measure the accuracy of a clustering method when we deal with data set without an a priori knowledge about class belonging ? (the data used for the clustering task, do not contain ...
4
votes
0answers
59 views

How to Validate a Monte Carlo Simulation

I have historical data of a production process, and I've being asked to build a simulation model to predict its performance in the future. Using the historical data, I've being able to obtain the ...
1
vote
3answers
61 views

Repeatedly split data in training (0.75) and test (0.25) for cross validation

What kind of cross validation is it called when we randomly split the data into 0.75 training and 0.25 test data set. And this split is done 1000 times.
1
vote
1answer
34 views

What is the proper name for “unknown data” set in machine learning?

As far as I know in practice the whole training set is usually split into training, validation and test[1] sets. Training set is used to train the model, validation to tune the parameters and test set ...
1
vote
0answers
12 views

What's the most appropriate way to derive and validate a model with hierarchical data

I am working on a model to predict the risk of some outcomes and could really use some advise: Let's say we have x number of patients, each patient have anywhere between 0 and y number of visits (...
0
votes
1answer
33 views

What are the alternatives to MMRE, PRED and MdMRE for validation?

I am working over the statistical validation of data. Till now I have computed MMRE, PRED and MdMRE. But I need alternatives to these because MRE is sensitive to data with large MRE's.
0
votes
0answers
36 views

What is the formula of Median Magnitude of Relative Error? (MdMRE)

I'm familiar with these terms 1 - MRE Mean Relative Error 2 - MMRE Mean Magnitude of Relative Error I need to compute MdMRE which is Median Mean Relative Error. I searched on the net but didn't ...
0
votes
0answers
13 views

Validating a qualitative method with quantitative data

I have developed an algorithm to detect micro events in sleep. These events have duration of a couple seconds, and in my data set each subject has around 100 of these for a full night being ~6-8 hours....
0
votes
0answers
10 views

How to split a survival data such that the proportion of events and censoring are equal in both groups

I need to develop a prognostic model, i have the survival data, and i need to split into validation set and training set. However, I want the Ratio of event to censoring in both sets to be equal. so ...
0
votes
1answer
17 views

How to address edge relations between training set and test set?

Suppose we are working on some sort of classification problem, and we have subdivided our data into a training set and a test set (or validation set, or etc.). We wish to prepare the data in the ...
0
votes
0answers
30 views

Why classification accuracy in validation set gets lower if validation cost also gets lower?

I'm training neural network for some simple classification task using tensorflow and have 2 output neurons, using softmax classification. My question is why accuracy on validation set gets lower when ...
0
votes
0answers
16 views

statistical tolerance

My request is: I have data (sums of money) and someone calculated tolerance limits for the data. I want to validate these tolerance limits with statistical methods/arguments. Does anyone know ...
4
votes
0answers
86 views

optimism-corrected regression coefficients using Frank Harrell's method?

I used a regularized (LASSO) cox regression to estimate relapse times of patients and used Frank Harrell's bootstrapping method to obtain an optimism-corrected performance estimate of my model. I am ...
0
votes
0answers
11 views

A new formula to estimate seminal vesicle volume? Which method to validate it?

I developed a new formula to estimate seminal vesicles volume, and tried it on 75 cases. I want to compare this new formula-method with golden standard method? I think i can use correlation(Pearson) ...
1
vote
0answers
56 views

How to validate a Poisson GLMM model?

I’m using the glmer function from the lme4 package in R to model species richness adjacent ...
0
votes
0answers
38 views

Self Organizing Map and input normalizing

I've been playing around with self organizing maps (SOM) recently. I tried to implement a simple example. You can see the training implementation function gist here and full contained SOM example ...
1
vote
0answers
25 views

ARIMA model over- or underfitting: compare training and validation performance

I'm doing research using seasonal and nonseasonal ARIMA models. Here's the result of model identification: Based on many sources, Your model is overfitting your training data when you see that ...
0
votes
0answers
21 views

Comparing a regression model with a single unit to the original model with all units

Background information: I have a regression model consisting of 230 companies (entity) for 20 years The model has 9 X-variables, and the P-value < 1% for the whole model The Y-...
1
vote
0answers
29 views

CNN training and overfitting

When testing the training of a CNN code with a small data set (approx 2560 images each for training and validation), what is over-fitting and how can it be mitigated? Arnold
0
votes
2answers
98 views

Model fitting: resampling the validation set to obtain distributions of test statistic

I see many descriptions of splitting the data set into a training part, a validation part and a test part. We train our models on the training part and choose the best model using the validation part, ...
1
vote
2answers
28 views

Validation set in presence of cross-validation

I am new to machine learning and want to ask regarding a confusion I have. I have a data set which is labeled and I want to do supervised learning. My question is related to cross-validation and ...
0
votes
0answers
27 views

why don't my CNN validation errors decrease?

I'm running theano_alexnet. I found that with greater numbers of iterations (20,000) the training cost and training error rates began to decrease (6.9 to 4.9 and 99% to 93%, respectively).When I ...
0
votes
0answers
36 views

R: Interpreting Fowlkes–Mallows index output for comparing dendrograms for hierarchical clustering

I have two data sets which contains information about subsystems in a bacterial metabolic model DataSet1: Behavior data of the subsystems DataSet2: Structural data of the same subsystems Then perform ...
0
votes
1answer
44 views

What should be validation strategy?

I am building CTR(https://en.wikipedia.org/wiki/Click-through_rate) Click prediction model with different (61) variables.Dependent variable is weather 0/1( click).I have build logistic regression ...
0
votes
2answers
185 views

Classification accuracy increasing while overfitting

I'm training a classification model, and these are the plots for accuracy and loss history. Besides the fact that the learning rate is too large, what I understand is that the model start ...
0
votes
1answer
27 views

validating model in machine learning - what does it mean in reality (intuition)

Could someone explain (in simple way) what does mean of validating model ? I tried to understand it, but I didn't managed to. I can do cross-validation, but I am not sure about if it is validation....
0
votes
0answers
12 views

How to summarize MSE across groups?

I have a large amount of time series data collected for different groups over a 30 year period (dataset x) that can be broken down by sub-group. and corresponding time series data (dataset y) from a ...
1
vote
0answers
21 views

When I can say a dataset is imbalanced?

I have a data set with only two outputs: positive and negative The ratio of positive:negative is 3.5:1 In this case, is my data set unbalanced? If so, what metric I should use to report the ...
1
vote
0answers
12 views

Does it make sense to use my in-sample data to look at goodness of fit for my model?

Basically, I came across an article where the authors first ran a logistic regression on a data set to predict the probability (q=demand) of buying their product, as a function of price p and various ...
0
votes
1answer
34 views

Topic Modeling Dataset for Code Verification

I am trying to write up a Gibbs sampling Latent Dirichlet Allocation function for myself in R, and wanted to run it on a dataset where the true classification of ...
1
vote
0answers
61 views

Cluster validation method for no cluster labels and differently sized clusters

I'm primarily a programmer and have little to no training in formal maths or statistics of any kind. I'm working on my dissertation (which foolishly is about clustering data), the process is ...
0
votes
0answers
38 views

Can I use the Xie-Beni index to validate data transformation parameters in fuzzy c-means clustering?

I am using fuzzy c-means algorithm to cluster my data in various feature spaces and the results differ depending on what kind of transformation I perform on my raw data. I want to know if using the ...
2
votes
1answer
59 views

Can we gain by merging validation and test set?

Reading this, Cross-validation including training, validation, and testing. Why do we need three subsets? I realized that if we can reduce the variance of the model performance, I wouldn't need the ...
0
votes
0answers
28 views

How to bootstrap validate regression model that involves removing outliers?

Suppose I have the following modelling process: Fit simple linear regression to whole data. Identify outliers, in the sense of having studentized residuals greater than a threshold, and remove them. ...