Questions tagged [validation]
The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.
865
questions
1
vote
1
answer
778
views
How to choose a model's hyperparameters in terms of the variance?
I was solving this question about tuning hyperparameters and I don't understand how to choose the number of hyperparameters by using the training error (TE) and the validation error (VE). Define the ...
-1
votes
1
answer
53
views
why random model response for X decile is fixed at X%?
I was reading online tutorials on lift and gain charts here, here, here
In all of these tutorials, I read or see that random model curve is drawn with the expectation at each Xth decile, we get X% ...
11
votes
1
answer
1k
views
How to calibrate models if we don't have enough data?
I am working on random forest classifiation with a dataset size of 977 records and 6 features. However, my class is imbalanced and proportion is 77:23
I was reading about calibration of models (binary ...
0
votes
0
answers
103
views
CART coverall accuracy vs. RF & SVM
I am performing a supervised classification with RF, SVM, and CART algorithms.
I have over 2000 training points in an area of 9,995 km².
For CART, I have obtained a 'Validation overall accuracy = 1' ...
3
votes
1
answer
274
views
How to plot the random and best model in Lift and gain charts?
I have been reading multiple tutorials on lift and gain charts here,here,here and here.
While I understand how the curve is drawn based on our model, but I don't know how the dotted black lines for ...
2
votes
0
answers
2k
views
Choosing the 'best' epoch to stop the training of neural network. Top accuracy not improving, but average is
I'm familiar with concepts like early stopping, and detection of plateau and so on.
Tensorflow CNN training has a possibility of saving only best model too, according to model's accuracy metric (for ...
2
votes
2
answers
58
views
Is "don't tune based on test" a small sample problem?
I am trying to wrap my head around some of the principles of Machine Learning, and in particular:
Why separate test and validation sets?
The error rate estimate of the final model on validation data ...
2
votes
0
answers
129
views
How to name train + validation set
Usually in machine learning pipelines, we use a train set, a validation set and a test set. Quite often, we first split the test set from the rest, and then we split the "rest" into train ...
2
votes
1
answer
2k
views
Why does XGBoost with cross-validation perform worse on test holdout than unvalidated model?
I have an XGBoost model that I fit on some X data directly out of the box:
...
1
vote
0
answers
105
views
Bootstrapped Latin Partition
I'm having trouble understanding the Bootstrapped Latin partition method (as presented in Statistical validation of classification and calibration models using bootstrapped Latin partitions and ...
2
votes
2
answers
782
views
Logistic regression metric
I am interested to understand in which scenarios person should use sensitivity, specificity, and when should person opt for precision recall.
On a high level I understand for a balanced data set we ...
1
vote
1
answer
319
views
Generalization of model performance (AUC) and tuning of a catboost classifier
I was wondering if it is good practice to overfit on the training data while tuning a catboost classifier for a binary outcome.
Wouldn't it be better to reguralize until validation error equals ...
5
votes
1
answer
426
views
Why custom evaluation/scoring metric is causing overfitting in (cross) validation?
I am using machine learning to approach a balanced binary classification task.
Some of rows are more important/valuable than others, so getting them right is extra important. Therefore, to accommodate ...
1
vote
0
answers
108
views
Why does one model overfits faster than another?
Lets' say we have 2 models and we train both of them on the same data for the same number of epochs. When we monitor the validation loss, we realize that the point where the validation loss stops ...
1
vote
0
answers
84
views
Purpose of validation set in model.fit?
What is the purpose of the validation set in model.fit(…)?
Does this touch the concept of cross validation at all? Afaik cross validation is done in an "...
2
votes
0
answers
76
views
Final predictive model: using all available data?
Objective: Build a screening tool to identify people at risk of X.
Approach: Using data from contexts A and B, we explored logistic regression models to predict X. We did forward & backward ...
1
vote
0
answers
50
views
Including the validation set in the training set
I observed this in a Kaggle M5 competition notebook (cell 7, some explanation in cell 4) that inspired the competition winner, who used the same methodology to create "fake" validation data. ...
2
votes
0
answers
109
views
What is the proper way to externally validate clusters when I have only a sample of the dataset labeled, but want to cluster the entire dataset?
I have a dataset of text-based documents that I want to cluster. For a sample of this dataset (~10%) I have manually annotated labels (i.e., the ground truth). I would like to cluster this dataset to &...
1
vote
0
answers
427
views
How does Walk-Forward work with LSTM
I have been looking at how to split my data for training/validation/test for a timeseries using LSTM and have some conflicting thoughts I would like to get a bit more clarity on.
I came across: QA1 ...
0
votes
1
answer
726
views
Data input: Expanding or Sliding Windows for LSTMs?
External research R1 (Stock Prediction with ML: Walk-forward Modeling by Chad Gray on 18/07/2018 at alphascientist.com) led me to believe that a sliding window is
more favourable than an expanding ...
4
votes
1
answer
5k
views
How to cross-validate a time series LSTM model?
I have been looking at how to split my data for training/validation/test for a timeseries using LSTM and came across: QA1 and QA2
Given I should implement walk-forward splits my depiction of it is:
...
1
vote
0
answers
15
views
What is possibly wrong in this Validation Accuracy and Training Accuracy [duplicate]
As general perception over training and validation accuracy is that if training accuracy is high and validation accuracy is marginally low, then it is most probably over fitting.
Consider a case of ...
2
votes
1
answer
54
views
Calculate predicted values after validation of logistic model
I have a simple logistic model, and I internally validated it using rms::validate in R. The estimated overoptimism for the intercept is -0.015 and for the slope is ...
0
votes
0
answers
43
views
Validation of ml-model with highly imbalanced labels
I'm currently working on a project which is facing highly imbalanced class labels. So let's say we have 1000 good labels and 5 bad labels. At the moment I'm not sure how long i have to collect data to ...
1
vote
1
answer
38
views
Psychometric test validation question
We are trying to validate a test for young children using IRT analyses (rasch/2pl). Models do not converge due to a floor effect. Teachers also took this assessment. Would it skew results if i ...
0
votes
0
answers
51
views
Test for a Bernoulli model with a single trial
I have a model that predicts Bernoulli probability parameters $p_i$ at $i\in[1..100]$ sites. To test this model, i can only take one trial at each one of the sites, resulting in $\approx10$ successes....
0
votes
0
answers
107
views
Statistical comparison of multiple methods on a single data set
I'm trying to optimize and evaluate multiple machine learning methods on a single data set. I know that the "usual" procedure with k-fold cross-validation works as follows:
Step: Split your ...
1
vote
0
answers
27
views
How can I validate my curve fitting meta-regression?
I'm doing a meta-regression analysis on Normal Pressure Hydrocephalus Gait Analysis. Studying the variation of gait velocity after a procedure (Tap-test). Keeping into account single study (#14) ...
2
votes
0
answers
40
views
Is testing/validation necessary when the model is not going to be used on new data?
I am doing an analysis on data where each row is a state, trying to predict energy consumption per capita from various other features. I am currently using LOOCV but I realized, what is the point of ...
2
votes
1
answer
4k
views
If my test size is small, should the validation set be the same size?
I know there is a rule of thumb to split the data to 70%-90% train data and 30%-10% validation data.
But if my test size is small, for example: its size is 5% of the size of the train set, and I can't ...
1
vote
1
answer
400
views
How do I report results of an internal validation in Caret?
I have the following question. In a machine learning project I have to solve a regression and a classification task. See also:
Hold-Out VS Cross-Validation - R caret
For this I have about ~650 cases ...
2
votes
0
answers
90
views
Validation loss falls but train loss remains constant? [closed]
My validation loss (left) falls to near 0, while my training loss (right) remains basically unchanged (gradient step is on the abscissa). This is the opposite of the typical error in which train loss ...
0
votes
1
answer
181
views
Can I calibrate to 100% of my sample in ML regression?
I have a standard ML regression model trained on 80% of my data with 20% saved for testing.
I want my model to match my full sample as best possible.
Can I multiply my outputs by mean(observations ...
0
votes
1
answer
325
views
Clarification about Cross Validation and Test Set
I was reading some questions and answer about the reasons and differences for the split of the datasets in Train, Validation and Test Set.
I came up with two questions that I'm not completely sure ...
2
votes
0
answers
126
views
Vertical Federated Learning
Lately I have been working (mostly reading) on Federated Learning and the one type of federation that looks suitable for my case is the Vertical Federated Learning. You can read about it here. Briefly,...
0
votes
0
answers
27
views
Is it possible to use training, validation and test split for regression models?
For classification problems, we can split the data into training, validation and test sets. And we can use the validation set for obtaining an unbiased estimation of the loss of our model. We can ...
1
vote
0
answers
34
views
Best way to train a model having train/test/val set
I'm new in the machine learning field and I'm trying to understand what are the best practices to manipulate the time series. I currently have a dataset structured as well:
X0: the first part of my ...
2
votes
1
answer
251
views
Cox proportional hazard Model $R^2 = 1$, good calibration - but Somers' $D = 0.16$
I have analyzed the survival time of a currency Unit "not being recovered" (for a debt) by fitting a Cox P. regression model.
Apparent $R^2=1$ is high, $D_{xy}=0.16$ . After bootstrap ...
0
votes
0
answers
28
views
How to decide model performance on validation data
I have built several neural networks to decide what choice of a hyperparameter is best. I want to use the validation data (not test data) to do this.
To determine the performance of each model should ...
0
votes
2
answers
875
views
How to decide model parameters of a neural network effectively
When choosing neural network parameters say numbers of features, layers and neurons, is the best way to do this by training each of the options several times by cross-validation and then take the ...
1
vote
1
answer
47
views
Why would you use a subset of the training set as the validation set in parametric classification?
I was told by my ML lecturer that the validation set, as used in parametric classification, is used to determine how overfitted the model is to the training data, but that it is also a subset of the ...
0
votes
2
answers
356
views
Random initialization of weights
I have trained a neural network using a train and a validation dataset.I used the validation dataset for hyperparameter and architecture optimization.I split the dataset in the exact same way each ...
0
votes
1
answer
87
views
Validating the percent variance explained of principal components on out-of-sample data
I'm trying to ascertain whether the variance explained by a certain PCA on an out-of-sample dataset is not due to random chance.
Suppose I have a dataset X with size n-x-p, and I run a PCA and obtain ...
5
votes
1
answer
850
views
Validation of a linear mixed effect Model
I am setting up models of the diversity data (Shannon Index) of bees and hoverflies to find out which in which studied seed mixture (site) the diversity was higher. In addition, I would like to know ...
3
votes
4
answers
3k
views
How many data points for test set in a time series
I have a monthly sales data set from 2018 January onwards. I would like to know from expert what is the optimum train test split and minimum train test split. Also to mention that my data includes ...
0
votes
1
answer
58
views
Two different types of K-Fold Cross-Validation. Are both ways right ? Advantages or disavantages of each?
For brevity let's say that k=10.
1º Scenario: I'll divide the training dataset in 10 parts of equal size, train my models in 9 groups joined together and measure some metrics on the remaining one, ...
1
vote
1
answer
34
views
How is validation of neural nets implemented?
I know what are training, validation and testing stages. But, I want to know how validation is implemented. Let's consider the following data : data_train, data_val, data_test as respectively training,...
0
votes
1
answer
150
views
Should models built using under-sampled data be evaluated against the population
I have a dataset of 11 mil. rows with a 1:10 ratio between minority and majority classes.
To train a model, I have selected all the minority class members and 1/3 of the majority class.
The ratio is ...
1
vote
0
answers
65
views
Validation of Clustering with labels
I am currently trying to perform clustering on 8 different datasets where I have 40-100 "labeled" data points per data set, representing which data points belong to the same cluster. I ...
2
votes
1
answer
52
views
Should one use the usual splitting (Learning/Validation/Test) when using cross-validation?
Say you want to tune several parameters of your model using $N$ data. What you usually do is splitting your $N$ data into 3 sets:
learning set: used to build your model;
validation set: used to ...