Questions tagged [validation]

The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.

Filter by
Sorted by
Tagged with
1 vote
1 answer
778 views

How to choose a model's hyperparameters in terms of the variance?

I was solving this question about tuning hyperparameters and I don't understand how to choose the number of hyperparameters by using the training error (TE) and the validation error (VE). Define the ...
Babado's user avatar
  • 111
-1 votes
1 answer
53 views

why random model response for X decile is fixed at X%?

I was reading online tutorials on lift and gain charts here, here, here In all of these tutorials, I read or see that random model curve is drawn with the expectation at each Xth decile, we get X% ...
The Great's user avatar
  • 3,272
11 votes
1 answer
1k views

How to calibrate models if we don't have enough data?

I am working on random forest classifiation with a dataset size of 977 records and 6 features. However, my class is imbalanced and proportion is 77:23 I was reading about calibration of models (binary ...
The Great's user avatar
  • 3,272
0 votes
0 answers
103 views

CART coverall accuracy vs. RF & SVM

I am performing a supervised classification with RF, SVM, and CART algorithms. I have over 2000 training points in an area of 9,995 km². For CART, I have obtained a 'Validation overall accuracy = 1' ...
user avatar
3 votes
1 answer
274 views

How to plot the random and best model in Lift and gain charts?

I have been reading multiple tutorials on lift and gain charts here,here,here and here. While I understand how the curve is drawn based on our model, but I don't know how the dotted black lines for ...
The Great's user avatar
  • 3,272
2 votes
0 answers
2k views

Choosing the 'best' epoch to stop the training of neural network. Top accuracy not improving, but average is

I'm familiar with concepts like early stopping, and detection of plateau and so on. Tensorflow CNN training has a possibility of saving only best model too, according to model's accuracy metric (for ...
silver_rocket's user avatar
2 votes
2 answers
58 views

Is "don't tune based on test" a small sample problem?

I am trying to wrap my head around some of the principles of Machine Learning, and in particular: Why separate test and validation sets? The error rate estimate of the final model on validation data ...
FZS's user avatar
  • 395
2 votes
0 answers
129 views

How to name train + validation set

Usually in machine learning pipelines, we use a train set, a validation set and a test set. Quite often, we first split the test set from the rest, and then we split the "rest" into train ...
Valérian Rey's user avatar
2 votes
1 answer
2k views

Why does XGBoost with cross-validation perform worse on test holdout than unvalidated model?

I have an XGBoost model that I fit on some X data directly out of the box: ...
NaiveBae's user avatar
  • 257
1 vote
0 answers
105 views

Bootstrapped Latin Partition

I'm having trouble understanding the Bootstrapped Latin partition method (as presented in Statistical validation of classification and calibration models using bootstrapped Latin partitions and ...
Effi T's user avatar
  • 11
2 votes
2 answers
782 views

Logistic regression metric

I am interested to understand in which scenarios person should use sensitivity, specificity, and when should person opt for precision recall. On a high level I understand for a balanced data set we ...
Teja Bandaru's user avatar
1 vote
1 answer
319 views

Generalization of model performance (AUC) and tuning of a catboost classifier

I was wondering if it is good practice to overfit on the training data while tuning a catboost classifier for a binary outcome. Wouldn't it be better to reguralize until validation error equals ...
infinite789's user avatar
5 votes
1 answer
426 views

Why custom evaluation/scoring metric is causing overfitting in (cross) validation?

I am using machine learning to approach a balanced binary classification task. Some of rows are more important/valuable than others, so getting them right is extra important. Therefore, to accommodate ...
Vladimir Belik's user avatar
1 vote
0 answers
108 views

Why does one model overfits faster than another?

Lets' say we have 2 models and we train both of them on the same data for the same number of epochs. When we monitor the validation loss, we realize that the point where the validation loss stops ...
lalaland's user avatar
  • 227
1 vote
0 answers
84 views

Purpose of validation set in model.fit?

What is the purpose of the validation set in model.fit(…)? Does this touch the concept of cross validation at all? Afaik cross validation is done in an "...
ManuelSchneid3r's user avatar
2 votes
0 answers
76 views

Final predictive model: using all available data?

Objective: Build a screening tool to identify people at risk of X. Approach: Using data from contexts A and B, we explored logistic regression models to predict X. We did forward & backward ...
jessexknight's user avatar
1 vote
0 answers
50 views

Including the validation set in the training set

I observed this in a Kaggle M5 competition notebook (cell 7, some explanation in cell 4) that inspired the competition winner, who used the same methodology to create "fake" validation data. ...
Roman's user avatar
  • 85
2 votes
0 answers
109 views

What is the proper way to externally validate clusters when I have only a sample of the dataset labeled, but want to cluster the entire dataset?

I have a dataset of text-based documents that I want to cluster. For a sample of this dataset (~10%) I have manually annotated labels (i.e., the ground truth). I would like to cluster this dataset to &...
BNise's user avatar
  • 21
1 vote
0 answers
427 views

How does Walk-Forward work with LSTM

I have been looking at how to split my data for training/validation/test for a timeseries using LSTM and have some conflicting thoughts I would like to get a bit more clarity on. I came across: QA1 ...
Panda's user avatar
  • 51
0 votes
1 answer
726 views

Data input: Expanding or Sliding Windows for LSTMs?

External research R1 (Stock Prediction with ML: Walk-forward Modeling by Chad Gray on 18/07/2018 at alphascientist.com) led me to believe that a sliding window is more favourable than an expanding ...
Panda's user avatar
  • 51
4 votes
1 answer
5k views

How to cross-validate a time series LSTM model?

I have been looking at how to split my data for training/validation/test for a timeseries using LSTM and came across: QA1 and QA2 Given I should implement walk-forward splits my depiction of it is: ...
Panda's user avatar
  • 51
1 vote
0 answers
15 views

What is possibly wrong in this Validation Accuracy and Training Accuracy [duplicate]

As general perception over training and validation accuracy is that if training accuracy is high and validation accuracy is marginally low, then it is most probably over fitting. Consider a case of ...
The Peace of Programming's user avatar
2 votes
1 answer
54 views

Calculate predicted values after validation of logistic model

I have a simple logistic model, and I internally validated it using rms::validate in R. The estimated overoptimism for the intercept is -0.015 and for the slope is ...
Claudio's user avatar
  • 155
0 votes
0 answers
43 views

Validation of ml-model with highly imbalanced labels

I'm currently working on a project which is facing highly imbalanced class labels. So let's say we have 1000 good labels and 5 bad labels. At the moment I'm not sure how long i have to collect data to ...
MB_1986's user avatar
1 vote
1 answer
38 views

Psychometric test validation question

We are trying to validate a test for young children using IRT analyses (rasch/2pl). Models do not converge due to a floor effect. Teachers also took this assessment. Would it skew results if i ...
em123's user avatar
  • 11
0 votes
0 answers
51 views

Test for a Bernoulli model with a single trial

I have a model that predicts Bernoulli probability parameters $p_i$ at $i\in[1..100]$ sites. To test this model, i can only take one trial at each one of the sites, resulting in $\approx10$ successes....
Anatoly's user avatar
  • 111
0 votes
0 answers
107 views

Statistical comparison of multiple methods on a single data set

I'm trying to optimize and evaluate multiple machine learning methods on a single data set. I know that the "usual" procedure with k-fold cross-validation works as follows: Step: Split your ...
Jannik's user avatar
  • 101
1 vote
0 answers
27 views

How can I validate my curve fitting meta-regression?

I'm doing a meta-regression analysis on Normal Pressure Hydrocephalus Gait Analysis. Studying the variation of gait velocity after a procedure (Tap-test). Keeping into account single study (#14) ...
Massimiliano 's user avatar
2 votes
0 answers
40 views

Is testing/validation necessary when the model is not going to be used on new data?

I am doing an analysis on data where each row is a state, trying to predict energy consumption per capita from various other features. I am currently using LOOCV but I realized, what is the point of ...
Ben's user avatar
  • 21
2 votes
1 answer
4k views

If my test size is small, should the validation set be the same size?

I know there is a rule of thumb to split the data to 70%-90% train data and 30%-10% validation data. But if my test size is small, for example: its size is 5% of the size of the train set, and I can't ...
Amit S's user avatar
  • 57
1 vote
1 answer
400 views

How do I report results of an internal validation in Caret?

I have the following question. In a machine learning project I have to solve a regression and a classification task. See also: Hold-Out VS Cross-Validation - R caret For this I have about ~650 cases ...
bckpex's user avatar
  • 61
2 votes
0 answers
90 views

Validation loss falls but train loss remains constant? [closed]

My validation loss (left) falls to near 0, while my training loss (right) remains basically unchanged (gradient step is on the abscissa). This is the opposite of the typical error in which train loss ...
Rylan Schaeffer's user avatar
0 votes
1 answer
181 views

Can I calibrate to 100% of my sample in ML regression?

I have a standard ML regression model trained on 80% of my data with 20% saved for testing. I want my model to match my full sample as best possible. Can I multiply my outputs by mean(observations ...
YuGiJoe's user avatar
0 votes
1 answer
325 views

Clarification about Cross Validation and Test Set

I was reading some questions and answer about the reasons and differences for the split of the datasets in Train, Validation and Test Set. I came up with two questions that I'm not completely sure ...
paolopazzo's user avatar
2 votes
0 answers
126 views

Vertical Federated Learning

Lately I have been working (mostly reading) on Federated Learning and the one type of federation that looks suitable for my case is the Vertical Federated Learning. You can read about it here. Briefly,...
s510's user avatar
  • 161
0 votes
0 answers
27 views

Is it possible to use training, validation and test split for regression models?

For classification problems, we can split the data into training, validation and test sets. And we can use the validation set for obtaining an unbiased estimation of the loss of our model. We can ...
Zaratruta's user avatar
  • 898
1 vote
0 answers
34 views

Best way to train a model having train/test/val set

I'm new in the machine learning field and I'm trying to understand what are the best practices to manipulate the time series. I currently have a dataset structured as well: X0: the first part of my ...
Fabio's user avatar
  • 115
2 votes
1 answer
251 views

Cox proportional hazard Model $R^2 = 1$, good calibration - but Somers' $D = 0.16$

I have analyzed the survival time of a currency Unit "not being recovered" (for a debt) by fitting a Cox P. regression model. Apparent $R^2=1$ is high, $D_{xy}=0.16$ . After bootstrap ...
Danny's user avatar
  • 1,035
0 votes
0 answers
28 views

How to decide model performance on validation data

I have built several neural networks to decide what choice of a hyperparameter is best. I want to use the validation data (not test data) to do this. To determine the performance of each model should ...
Gabi23's user avatar
  • 143
0 votes
2 answers
875 views

How to decide model parameters of a neural network effectively

When choosing neural network parameters say numbers of features, layers and neurons, is the best way to do this by training each of the options several times by cross-validation and then take the ...
Gabi23's user avatar
  • 143
1 vote
1 answer
47 views

Why would you use a subset of the training set as the validation set in parametric classification?

I was told by my ML lecturer that the validation set, as used in parametric classification, is used to determine how overfitted the model is to the training data, but that it is also a subset of the ...
jess's user avatar
  • 113
0 votes
2 answers
356 views

Random initialization of weights

I have trained a neural network using a train and a validation dataset.I used the validation dataset for hyperparameter and architecture optimization.I split the dataset in the exact same way each ...
Epitheoritis 32's user avatar
0 votes
1 answer
87 views

Validating the percent variance explained of principal components on out-of-sample data

I'm trying to ascertain whether the variance explained by a certain PCA on an out-of-sample dataset is not due to random chance. Suppose I have a dataset X with size n-x-p, and I run a PCA and obtain ...
JSL's user avatar
  • 1
5 votes
1 answer
850 views

Validation of a linear mixed effect Model

I am setting up models of the diversity data (Shannon Index) of bees and hoverflies to find out which in which studied seed mixture (site) the diversity was higher. In addition, I would like to know ...
Manuel Schulz's user avatar
3 votes
4 answers
3k views

How many data points for test set in a time series

I have a monthly sales data set from 2018 January onwards. I would like to know from expert what is the optimum train test split and minimum train test split. Also to mention that my data includes ...
Prasenjit Datta's user avatar
0 votes
1 answer
58 views

Two different types of K-Fold Cross-Validation. Are both ways right ? Advantages or disavantages of each?

For brevity let's say that k=10. 1º Scenario: I'll divide the training dataset in 10 parts of equal size, train my models in 9 groups joined together and measure some metrics on the remaining one, ...
Fernando's user avatar
1 vote
1 answer
34 views

How is validation of neural nets implemented?

I know what are training, validation and testing stages. But, I want to know how validation is implemented. Let's consider the following data : data_train, data_val, data_test as respectively training,...
narutoArea51's user avatar
0 votes
1 answer
150 views

Should models built using under-sampled data be evaluated against the population

I have a dataset of 11 mil. rows with a 1:10 ratio between minority and majority classes. To train a model, I have selected all the minority class members and 1/3 of the majority class. The ratio is ...
onejerlo's user avatar
1 vote
0 answers
65 views

Validation of Clustering with labels

I am currently trying to perform clustering on 8 different datasets where I have 40-100 "labeled" data points per data set, representing which data points belong to the same cluster. I ...
S.MC.'s user avatar
  • 111
2 votes
1 answer
52 views

Should one use the usual splitting (Learning/Validation/Test) when using cross-validation?

Say you want to tune several parameters of your model using $N$ data. What you usually do is splitting your $N$ data into 3 sets: learning set: used to build your model; validation set: used to ...
Akusa's user avatar
  • 331

1 2
3
4 5
18