Validation is the process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing `validity` of a measurement or instrument -- such as that it measures what it purports to.

learn more… | top users | synonyms

0
votes
2answers
153 views

highly sporadic validation error during training with multilayer perceptron

I'm encountering an issue where a classifier I'm developing reports validation errors during training that span a wide range of values without consistently decreasing over time. Unfortunately, I'm new ...
0
votes
0answers
4 views

Add Structure on Experimental semivariogram and validation of semivariogram [on hold]

im fairly new to this so please be patient with me. I am using surpac 6.3.2 and i want to know when to add a "structure" to the experimental semivar, also, once i have completed a semivar, how to i ...
0
votes
0answers
16 views

how to measure clustering task with unlabelled data set [duplicate]

I wanna know, how to measure the accuracy of a clustering method when we deal with data set without an a priori knowledge about class belonging ? (the data used for the clustering task, do not contain ...
4
votes
0answers
53 views

How to Validate a Monte Carlo Simulation

I have historical data of a production process, and I've being asked to build a simulation model to predict its performance in the future. Using the historical data, I've being able to obtain the ...
-1
votes
0answers
21 views

cluster external validation [on hold]

I am using ELKI in order to perform location clustering with DBSCAN and OPTICS. My data set include 30 participants but it is not labeled but I do have pair of coordinates (e.g. home, work, etc) as ...
3
votes
1answer
208 views

Minimal number of samples/conversions for statistical validity

We are measuring conversion rates (% of visitors who bought) on an e-commerce site. The test apply to a segment of visitors who meet specific criteria (for example people from a certain country). ...
1
vote
2answers
51 views

Repeatedly split data in training (0.75) and test (0.25) for cross validation

What kind of cross validation is it called when we randomly split the data into 0.75 training and 0.25 test data set. And this split is done 1000 times.
5
votes
1answer
489 views

Prediction evaluation metric for panel/longitudinal data

I would like to evaluate several different models that provide predictions of behavior at a monthly level. The data is balanced, and $n=$100,000 and $T=$12. The outcome is attending a concert in a ...
1
vote
1answer
32 views

What is the proper name for “unknown data” set in machine learning?

As far as I know in practice the whole training set is usually split into training, validation and test[1] sets. Training set is used to train the model, validation to tune the parameters and test set ...
4
votes
0answers
81 views

optimism-corrected regression coefficients using Frank Harrell's method?

I used a regularized (LASSO) cox regression to estimate relapse times of patients and used Frank Harrell's bootstrapping method to obtain an optimism-corrected performance estimate of my model. I am ...
2
votes
2answers
2k views

How to plot AIC values when using the leaps package?

Does anybody know how to plot all AIC values for different size models, when using the command regsubsets from the package leaps?...
4
votes
2answers
625 views

Logistic regression performs better on validation data

Recently I've been building a model using logistic regression. To my suprisise LIFT chart looks better on the validation data than on the training data, the same is with ROC. All variables in the ...
1
vote
0answers
9 views

What's the most appropriate way to derive and validate a model with hierarchical data

I am working on a model to predict the risk of some outcomes and could really use some advise: Let's say we have x number of patients, each patient have anywhere between 0 and y number of visits (...
3
votes
1answer
253 views

Validation of a questionnaire in a new population

I have 400 responses to a 20 item questionnaire which purports to measure an attitudinal constuct in medical students. The instrument was validated in the US for a single year of medical students and ...
0
votes
1answer
31 views

What are the alternatives to MMRE, PRED and MdMRE for validation?

I am working over the statistical validation of data. Till now I have computed MMRE, PRED and MdMRE. But I need alternatives to these because MRE is sensitive to data with large MRE's.
0
votes
0answers
25 views

What is the formula of Median Magnitude of Relative Error? (MdMRE)

I'm familiar with these terms 1 - MRE Mean Relative Error 2 - MMRE Mean Magnitude of Relative Error I need to compute MdMRE which is Median Mean Relative Error. I searched on the net but didn't ...
4
votes
2answers
2k views

Using Adaboost for feature selection?

Is it okay to use Adaboost to do feature selection (selecting a subset of dimensions $S$ from a high-dimensional feature vector $V$)? I divided the samples into four non-overlapping sets: $A$ (...
0
votes
0answers
13 views

Validating a qualitative method with quantitative data

I have developed an algorithm to detect micro events in sleep. These events have duration of a couple seconds, and in my data set each subject has around 100 of these for a full night being ~6-8 hours....
0
votes
0answers
10 views

How to split a survival data such that the proportion of events and censoring are equal in both groups

I need to develop a prognostic model, i have the survival data, and i need to split into validation set and training set. However, I want the Ratio of event to censoring in both sets to be equal. so ...
0
votes
1answer
17 views

How to address edge relations between training set and test set?

Suppose we are working on some sort of classification problem, and we have subdivided our data into a training set and a test set (or validation set, or etc.). We wish to prepare the data in the ...
0
votes
0answers
24 views

Why classification accuracy in validation set gets lower if validation cost also gets lower?

I'm training neural network for some simple classification task using tensorflow and have 2 output neurons, using softmax classification. My question is why accuracy on validation set gets lower when ...
6
votes
1answer
1k views

Within-group sum of squares of cluster

I have a multivariate dataset for which I have only a table including the cross-wise Euclidean distances between all points and a list giving the assignment of each point to one of several clusters. ...
0
votes
0answers
16 views

statistical tolerance

My request is: I have data (sums of money) and someone calculated tolerance limits for the data. I want to validate these tolerance limits with statistical methods/arguments. Does anyone know ...
2
votes
1answer
105 views

The effect of oversampling on the positive predictive value

I need to calculate the positive predictive value for a validation set for a rare event. The problem is that the validation set was oversampled for the rare event. The event occurs in 5 percent of the ...
1
vote
0answers
23 views

ARIMA model over- or underfitting: compare training and validation performance

I'm doing research using seasonal and nonseasonal ARIMA models. Here's the result of model identification: Based on many sources, Your model is overfitting your training data when you see that ...
0
votes
0answers
10 views

A new formula to estimate seminal vesicle volume? Which method to validate it?

I developed a new formula to estimate seminal vesicles volume, and tried it on 75 cases. I want to compare this new formula-method with golden standard method? I think i can use correlation(Pearson) ...
1
vote
0answers
42 views

How to validate a Poisson GLMM model?

I’m using the glmer function from the lme4 package in R to model species richness adjacent ...
0
votes
0answers
22 views

Self Organizing Map and input normalizing

I've been playing around with self organizing maps (SOM) recently. I tried to implement a simple example. You can see the training implementation function gist here and full contained SOM example ...
0
votes
0answers
21 views

Comparing a regression model with a single unit to the original model with all units

Background information: I have a regression model consisting of 230 companies (entity) for 20 years The model has 9 X-variables, and the P-value < 1% for the whole model The Y-...
0
votes
2answers
97 views

Model fitting: resampling the validation set to obtain distributions of test statistic

I see many descriptions of splitting the data set into a training part, a validation part and a test part. We train our models on the training part and choose the best model using the validation part, ...
1
vote
0answers
19 views

CNN training and overfitting

When testing the training of a CNN code with a small data set (approx 2560 images each for training and validation), what is over-fitting and how can it be mitigated? Arnold
1
vote
2answers
27 views

Validation set in presence of cross-validation

I am new to machine learning and want to ask regarding a confusion I have. I have a data set which is labeled and I want to do supervised learning. My question is related to cross-validation and ...
0
votes
0answers
21 views

why don't my CNN validation errors decrease?

I'm running theano_alexnet. I found that with greater numbers of iterations (20,000) the training cost and training error rates began to decrease (6.9 to 4.9 and 99% to 93%, respectively).When I ...
0
votes
0answers
31 views

R: Interpreting Fowlkes–Mallows index output for comparing dendrograms for hierarchical clustering

I have two data sets which contains information about subsystems in a bacterial metabolic model DataSet1: Behavior data of the subsystems DataSet2: Structural data of the same subsystems Then perform ...
0
votes
1answer
42 views

What should be validation strategy?

I am building CTR(https://en.wikipedia.org/wiki/Click-through_rate) Click prediction model with different (61) variables.Dependent variable is weather 0/1( click).I have build logistic regression ...
0
votes
2answers
99 views

Classification accuracy increasing while overfitting

I'm training a classification model, and these are the plots for accuracy and loss history. Besides the fact that the learning rate is too large, what I understand is that the model start ...
0
votes
1answer
25 views

validating model in machine learning - what does it mean in reality (intuition)

Could someone explain (in simple way) what does mean of validating model ? I tried to understand it, but I didn't managed to. I can do cross-validation, but I am not sure about if it is validation....
0
votes
0answers
11 views

How to summarize MSE across groups?

I have a large amount of time series data collected for different groups over a 30 year period (dataset x) that can be broken down by sub-group. and corresponding time series data (dataset y) from a ...
1
vote
0answers
20 views

When I can say a dataset is imbalanced?

I have a data set with only two outputs: positive and negative The ratio of positive:negative is 3.5:1 In this case, is my data set unbalanced? If so, what metric I should use to report the ...
1
vote
0answers
11 views

Does it make sense to use my in-sample data to look at goodness of fit for my model?

Basically, I came across an article where the authors first ran a logistic regression on a data set to predict the probability (q=demand) of buying their product, as a function of price p and various ...
0
votes
1answer
31 views

Topic Modeling Dataset for Code Verification

I am trying to write up a Gibbs sampling Latent Dirichlet Allocation function for myself in R, and wanted to run it on a dataset where the true classification of ...
1
vote
0answers
57 views

Cluster validation method for no cluster labels and differently sized clusters

I'm primarily a programmer and have little to no training in formal maths or statistics of any kind. I'm working on my dissertation (which foolishly is about clustering data), the process is ...
0
votes
0answers
37 views

Can I use the Xie-Beni index to validate data transformation parameters in fuzzy c-means clustering?

I am using fuzzy c-means algorithm to cluster my data in various feature spaces and the results differ depending on what kind of transformation I perform on my raw data. I want to know if using the ...
2
votes
1answer
58 views

Can we gain by merging validation and test set?

Reading this, Cross-validation including training, validation, and testing. Why do we need three subsets? I realized that if we can reduce the variance of the model performance, I wouldn't need the ...
0
votes
0answers
23 views

How to bootstrap validate regression model that involves removing outliers?

Suppose I have the following modelling process: Fit simple linear regression to whole data. Identify outliers, in the sense of having studentized residuals greater than a threshold, and remove them. ...
0
votes
0answers
17 views

External validation of a group's machine learning process

Say there's a group (that is opaque to you) that's heavily using machine learning methods to produce some outputs. You don't have access to their input data or code but can ask high level questions ...
0
votes
0answers
13 views

What sort of cross validation is this?

I've always tested my classification techniques using non-standardised trial and error but I'm interested to see which category my techniques fall under. They seem to fall under several but I'm not ...
0
votes
2answers
23 views

What do we learn from a test set?

Suppose I split my data into two parts -- a training set (having 80% of my data) and a testing (20%) set. I train a model on my training set, and test it on the test set. What do we learn from ...
1
vote
0answers
3 views

Validation of prediction interval for count data

I have developed a random-effects (frailty) survival model for repeated events which enables calculating individual-specific mean and prediction interval for the cumulative incidence (rate) of future ...