Questions tagged [validation]

The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.

Filter by
Sorted by
Tagged with
1
vote
1answer
20 views

threshold choice for binary classifier: on training, validation or test set?

I have a binary classification problem where I perform cross validation on the training set (currently 80% of the examples) and then evaluate results on a test set. I use cross validation for finding ...
0
votes
0answers
15 views

Comparing validity measure of K Means PAM DBSCAN and Fuzzy K Means

I analyze the cancellation of shopping baskets of different types of customers using cluster methods. My data set contains 821,000 entries and 9 variables. I have used the cluster procedures K Means, ...
1
vote
0answers
46 views

How to model non-experimental continuous individual treatment effect on a binary target?

I have historical data of a problem that can be described as: A person represented by features X has to wait T minutes on a queue so she can receive a treatment (which is equal for everyone), and ...
0
votes
0answers
7 views

Why do my internal and external validation metrics contradict each other?

I am currently doing an exhaustive feature selection for a clustering analysis in which I am testing every single combination of features, and then calculating internal and external validation metrics ...
0
votes
0answers
8 views

How to validate a confirmatory k-means cluster analysis in SPSS?

I've conducted a k-means cluster analysis in SPSS using the Z scores of two continuous variables for which the number of clusters was known a priori and the total number of observations exceeded 2000. ...
0
votes
0answers
10 views

Keras Validation of binary classifciation not the same as manual calculated

i have a Keras Sequential Network and use the validation_data attribute to get an idea of the validation straight away. However when the model is trained and I predict on the validation set I dont ...
0
votes
0answers
15 views

Statistical Comparison of Models with Multiple Performance Criteria

The Setup: I am trying to determine if two different methodologies have a statistically significant difference in performance. I have 15 different datasets, and each dataset has a training period, ...
1
vote
0answers
19 views

Statistical test to compare predictive measures with count data

I'm reviewing a paper that compares predictors of healthcare demand in terms of hospital visits for several conditions. The authors have used a Pearson's rho to determine which is best, and further ...
0
votes
0answers
12 views

When performing validation, should you also tune the number of hidden units and hidden layers?

I understand tuning the learning rate, momentum, batch size, etc. and finding the best set of parameters using the validation set. However, I don't understand when people say that you should also ...
1
vote
0answers
23 views

What method should I use to cluster small data set?

I would like to cluster a small data sets [23 genes, 50 samples], but I don't know what method I should use... could you give me any recommendation? I have applied hierarchical clustering (Wards ...
0
votes
1answer
22 views

Perceptual Loss Layers Selection

I understand that in order to improve your generative model performance it is quite useful to compare your output and the target in the feature space, as stated in the paper Perceptual Losses for Real-...
0
votes
0answers
38 views

Machine Learning: Why do I have this pattern of train and validation accuracy?

I am trying to understand what would generate this pattern of accuracy in train and validation dataset (second and third plot below). I am training a network to recognize 6 types of faces (they are ...
1
vote
0answers
11 views

Validation of Models with Different Scale Data [duplicate]

I created a model for two different datasets that have different scales. When checking which one performed better, I am struggling with figuring out the best methodology. My top choices right now are <...
2
votes
1answer
16 views

Stability Index for Fuzzy c Means Clustering in R

I am looking for a stability measure/index for Fuzzy C Means Clustering in R. Can anyone direct me to a package or a code in R? I am not looking for internal indices, only stability measures.
0
votes
0answers
27 views

What causes the high OOB-error for randomForest() in R?

I'm trying to perform a random forest in R on a dataset with 16364 observations (after undersampling), using the function randomForest(). But my results look really weird: What could have caused this?...
0
votes
0answers
51 views

Linear regression <> survival models in R

I have a question about survival analysis with a gaussian distribution of the response variable. I have interval censored data, that look something like this (just an example): ...
1
vote
0answers
30 views

Forecasting Validation

question on validating forecast models. The traditional approach (https://otexts.com/fpp2/forecasting-on-training-and-test-sets.html) is to hold-out n number of samples from the end of your time ...
2
votes
2answers
41 views

Do I need a validation set?

Do I need a validation set if I am using cross validation and grid search for parameter tuning? Similar to this question - what I understood is that it helps to prevent overfitting but it's not ...
1
vote
1answer
24 views

A meaning/significance of validation loss in a Generative Adversarial Neural Network? [closed]

On most of the tutorials on GANs that I came across the only monitored quantity is training loss. 1) Are there any general conclusions that could be derived from comparing training and validation ...
1
vote
1answer
24 views

Validation and Test accuracy at random performance, whereas Train accuracy very high

I am trying to build a classifier in TensorFlow2.1 for CIFAR10 using ResNet50 pre-trained over imagenet from keras.application and then stacking a small FNN on top of it: ...
1
vote
0answers
16 views

Test Time Augmentation on Validation set?

In the traditional usage of data augmentations, we augment only the train set examples, in order to keep the distribution of the validation and test set equal. In the TTA method, we apply ...
0
votes
0answers
10 views

How to measure ranking accuracy of CORD-19 search engine

Many of you might have heard that the Allen Institute for AI released a large text corpus called CORD-19 for text mining purposes. This corpuses is simply speaking just 60k corona papers in JSON ...
0
votes
0answers
13 views

Resampling Ground Truth - Manipulation?

A validation task requires comparing a timebound-signal (out of a system under test) of length $1\times m$ to be compared against a ground truth (GT; reference) timebound-signal of length $1\times n$, ...
1
vote
0answers
24 views

eval_set in xgboost and validation data

In my understanding, if I am picking from a set of models, each with a different set of hyper-parameters, the proper way to approach it is like this: First, split the data to ...
0
votes
0answers
9 views

Can you have a concordance statistic (discrimination) when predicting a continuous outcome?

I'm writing a proposal for a prediction model predicting BP (continuous outcome, predicting trend over time). For assessing model performance, I'm seeing discrimination and calibration as the most ...
1
vote
0answers
9 views

What is the difference between validation and verification in statistical models?

We assume errors in our models are additive and uncorrelated, this is a big hypothesis but we'll assume that it is true. This means that observations have an inherent error $\epsilon_{o}$. ...
0
votes
0answers
14 views

How K-Fold Cross-Validation will help in avoiding consumption of training data set?

One way of hyperparameter tuning is to try different values on validation test, suppose we divide the data into training(60%), validation(20%) and test(20%) set. Here disadvantage is we are eating up ...
1
vote
2answers
22 views

Is resampling to obtain a test set ok?

I have a question about cross-validation and hyperparameter optimization. Namely about how to test the final model performance in an unbiased way. Now, I know I have to train the model, optimize the ...
1
vote
1answer
18 views

Validation loss decreasing faster than training loss

I have two different scenarios that I ran across and I can't seem to wrap my head around what caused them. In this scenario, my validation loss, in orange, initially fell faster than my training loss,...
0
votes
0answers
11 views

Is it needed to validate Pearson R² or Spearman's rho?

I was wondering if, in the case of Pearson's correlation or Spearman's one, there is a need to validate the results beyond the p-value. Is it needed to test the residuals' normality or that there are ...
0
votes
0answers
4 views

Terminology: Hillclimb (validation) set - is this the same as a hold-out set?

The paper Ensemble Selection from Libraries of Models (Caruana et al, 2004) refers to "a hillclimb (validation) set". My questions are: It this what I know as a 'hold-out validation' set, i.e. a set ...
0
votes
1answer
15 views

Validation with only one dataset

I am performing multiple linear regression and multivariate analyses on a dataset (~600 samples). I would ideally use a second, independent dataset to validate my findings, but there is no suitable ...
1
vote
0answers
34 views

Model Validation Using Residual Plot?

I am trying to validate a given multiple linear regression model on a new dataset (a dataset that has not been "seen" by this model). Is it okay to discard this model on account of it showing a ...
1
vote
1answer
37 views

Keras: What is the meaning of batch_size for validation?

If I understand correctly that batch size is the number of samples used in the training of a NN before the gradient gets updated, then why do we need a specified batch_size for the validation sample? ...
0
votes
1answer
10 views

Is this a valid factor analysis approach for validating a multi-domain survey?

I am developing a new instrument consisting of items grouped into defined domains (constructs), such as personality traits. Each construct is measured through a set of 4-6 items (questions). To ...
1
vote
1answer
25 views

When working with unbalanced data, do we train final model on full data set?

Let's say we have an unbalanced data set. We randomly sample an amount from our larger class so that we have a balanced data set. After tuning parameters/hyperparameters and determining which ...
1
vote
1answer
21 views

categorical distribution in validation set

I have a dataset that contains 6877 samples. This is a multiclass multilabel classification which means that we have 9 classes and every sample can belong to one or more of these classes. The total ...
0
votes
0answers
9 views

Can someone explain how data leakage occurs if you randomize time series data?

I am a bit dense and I am not understanding the intuition behind why data leakage would occur if we don't "respect" the temporal order of time series data when doing sample splitting or cross ...
0
votes
0answers
14 views

Bias-Variance tradeoff in validation set approach

I read that validation set approach results in high bias because it tends to overestimate the test error rate. According to the Bias-Variance trade-off, does that mean this approach would have low ...
0
votes
0answers
24 views

test accuracy is too low compared toTraining and Validation accuracy for resampled data

I am working on https://www.kaggle.com/c/sentiment-analysis-on-movie-reviews dataset. The dataset is imbalance so I resample it (code mentioned below) For 20 epochs with Conv1d model (Keras), I get ...
0
votes
0answers
18 views

RNN test error calculation: teacher forcing or prediction data?

RNN training is mostly done by a teacher forcing method: [Training] y_{pred}^{i+1} = RNN(y_{real}^{i}) [Training error] Error_{train} = abs(y_{real}^{i+1} - y_{pred}^{i+1}) However, RNN applications ...
1
vote
1answer
98 views

In sample splitting for time series data, do we randomly select data?

I'm having a hard to conceptually understanding how to do this. I would like to do my own sample splitting (not the method built into a package). Let's say you have 80 days of weather data. You ...
0
votes
1answer
54 views

How to make validation curve more smooth?

I have this validation curve where in for each epoch I plot my validation score and training score. I use the Adam optimizer and don't attain better smoothness when I lower my learning rate, only when ...
0
votes
0answers
11 views

Good Book suggestions for model validation statistical tests? (Gini, Kolmogorov-Smirnov, Binomial/Adjusted binomial tests, Chi-square)

Any good books suggestion for the above mentioned topics such as Gini, Kolmogorov-Smirnov, Binomial/Adjusted binomial tests, Chi-square etc. tests?
0
votes
0answers
12 views

Model selection: testing accuracy higher than cross-validation

I trained convnets with different architetures and validated using 5-fold cross-validation. After this, I trained the final version of the convnets using the full data set (training + validation ...
0
votes
2answers
33 views

Validation criterias in simple linear regression in r

I plotted a simple linear regression in ggplot both for the entire data and training set and test set. But, the problem is that it just provides a plot, nothing more like RMSE, R2, etc. How can I ...
2
votes
0answers
19 views

Any methods for sanity checks on refreshed machine learning models?

I am wondering if there are any best practices for validating ML models that are trained on new data. Apart from the validation metrics used on test data, are there any other recommended approaches ...
0
votes
0answers
24 views

compare two GLM models on test set using deviance reduction with offset prediction

I'm trying to compare the goodness of fit of two GLM models of claims frequency, one "baseline" and one with some more variables we recovered from external sources. The approach I'm following should ...
0
votes
0answers
16 views

Why is out of bag error higher than out of fold error?

There are two reasosns that i have come across for out of bag error being higher than the expected OOF error. these are: Out of bag predictions are on a subset of the forest for which a particular ...
0
votes
0answers
28 views

Quantifying difference in model performance?

Probably a very basic question that's been asked before, but ... I have a set of candidate models (mixed-effects linear regressions) I'm fitting to 1/0 (presence/absence) data in R using ...

1
2 3 4 5
14