Questions tagged [validation]

The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

Validation Loss for my binary image classifier model is increasing. how to bring it down? [duplicate]

I am new to the domain of Deep learning and I have been trying to create a binary image classifier using a dataset which I created by myself. I am building the model from scratch. It is CNN model. ...
user avatar
  • 1
1 vote
0 answers
10 views

How do I perform a train-validation split on data with class imbalance such that the class imbalance ratio is preserved?

My data has class imbalance-- that is, some classes have significantly fewer training samples than the others. I want to perform a train-validation split in such as way that the class ratios are ...
user avatar
0 votes
0 answers
25 views

Cluster Validation and Sample Size [closed]

I've recently been using the aweSOM package in R https://cran.r-project.org/web/packages/aweSOM/vignettes/aweSOM.html. The objective is to generate a SOM then impose partitive clustering on top, in ...
user avatar
  • 174
0 votes
0 answers
8 views

when we lacked of some colums in validation set, how to use validation set in regression problem?

I am doing the project related with regression problem like failure rate prediction. For training data set, I have columns like below I have 10 columns training data. And for validation set, it is ...
user avatar
0 votes
0 answers
7 views

Cluster Validation on SOM Codebook

I've recently been using the aweSOM R package for cluster visualisation, https://cran.r-project.org/web/packages/aweSOM/vignettes/aweSOM.html. In particular, the aweSOM package entails using ...
user avatar
  • 174
0 votes
0 answers
17 views

Comparison of normality + similarity distribution of time series datas

I have a training time series data , whose last 20% is kept as validation data. I want to check whether the distribution of training and validation features are similiar and normal, so that we can say ...
user avatar
  • 1
0 votes
0 answers
7 views

The validation set loss first decreased and then increased

hello everyone: For multimodal sentiment classification task, the loss function is NLLLoss(weight=weight, reduction='sum'), loss(pred*mask_, target) /torch.sum(self.weight[target]*mask_.squeeze()) ...
user avatar
1 vote
0 answers
15 views

Validation accuracy does not reflect the actual change in the testing accuracy

When training a CNN classifier for the Fashion-MNIST, I have noticed that there are multiple instances that validation error is not improving even if the tesing accuracy is improving every epoch or ...
user avatar
  • 287
0 votes
0 answers
16 views

An explanation of swap set analysis and swap set table

I'm trying to understand swap set analysis and the way results from such an analysis are presented. This analysis is a way to compare two models to determine which one is better. The results are ...
user avatar
  • 141
1 vote
0 answers
14 views

In the training of a neural network, the validation loss increase and then decrease with the trainning loss?

When i train a MLP to learn a data map relationship, but i find behavior of the validation loss is very weird. The validation loss increase first and then it decrease with the trainning losss. But the ...
user avatar
0 votes
0 answers
14 views

What does it mean if the validation accuracy is equal to the testing accuracy?

I am training a CNN model for my specific problem. I have divided the dataset into 70% training set, 20% validation set, and 10% test set. The validation accuracy achieved was 95% and the test ...
user avatar
  • 13
0 votes
0 answers
8 views

SMALL SAMPLE SIZE: Can i use it for a validation study? What evidence supports this?

I would like to validate submaximal exercise tests; I will correlate the predicted maximum oxygen consumption values using the field tests (values will be derived from standard equation) with the ...
user avatar
  • 1
1 vote
1 answer
16 views

Validation supervised model [closed]

This is my first approact to Machine Learning. I want to know if there are other methods besides train, validation and test. Thanks.
user avatar
1 vote
1 answer
27 views

Issues regarding cross-validation and metrics for comparing approaches in machine learning for image classification with imbalanced datasets

I'm trying to compare the performances of N classifiers for multiclass image classification, with an imbalanced dataset with 50 classes. I'm considering now the following basic metrics: accuracy: ...
user avatar
  • 378
4 votes
3 answers
270 views

Practical justification for not basing model selection on performance on test data?

I've always been told not to use a model's performance on the test data to select a final model. I've read the responses to this question and others posted around the internet, but still have ...
user avatar
1 vote
2 answers
59 views

Kolmogorov-Smirnov p-value and alpha value in python

I am having trouble understanding the scipy.stats documentation against the backdrop of a book that explains the KS-Test. The documentation (https://docs.scipy.org/doc/scipy/reference/generated/scipy....
user avatar
1 vote
1 answer
36 views

What part of a dataset do I apply a traditional, statistical analysis to linear regression?

Note: I've edited my question as recommended below by @EdM. Suppose I have a supervised learning problem on a sizeable tidy dataset with real values—-e.g., the dataset has 100,000 rows or observations....
user avatar
1 vote
1 answer
89 views

How to choose a model's hyperparameters in terms of the variance?

I was solving this question about tuning hyperparameters and I don't understand how to choose the number of hyperparameters by using the training error (TE) and the validation error (VE). Define the ...
user avatar
  • 111
-1 votes
1 answer
25 views

why random model response for X decile is fixed at X%?

I was reading online tutorials on lift and gain charts here, here, here In all of these tutorials, I read or see that random model curve is drawn with the expectation at each Xth decile, we get X% ...
user avatar
  • 1,716
8 votes
1 answer
279 views

How to calibrate models if we don't have enough data?

I am working on random forest classifiation with a dataset size of 977 records and 6 features. However, my class is imbalanced and proportion is 77:23 I was reading about calibration of models (binary ...
user avatar
  • 1,716
0 votes
0 answers
36 views

CART coverall accuracy vs. RF & SVM

I am performing a supervised classification with RF, SVM, and CART algorithms. I have over 2000 training points in an area of 9,995 km². For CART, I have obtained a 'Validation overall accuracy = 1' ...
user avatar
2 votes
1 answer
31 views

How to plot the random and best model in Lift and gain charts?

I have been reading multiple tutorials on lift and gain charts here,here,here and here. While I understand how the curve is drawn based on our model, but I don't know how the dotted black lines for ...
user avatar
  • 1,716
0 votes
0 answers
39 views

Choosing the 'best' epoch to stop the training of neural network. Top accuracy not improving, but average is

I'm familiar with concepts like early stopping, and detection of plateau and so on. Tensorflow CNN training has a possibility of saving only best model too, according to model's accuracy metric (for ...
user avatar
2 votes
2 answers
41 views

Is "don't tune based on test" a small sample problem?

I am trying to wrap my head around some of the principles of Machine Learning, and in particular: Why separate test and validation sets? The error rate estimate of the final model on validation data ...
user avatar
  • 183
2 votes
0 answers
34 views

How to name train + validation set

Usually in machine learning pipelines, we use a train set, a validation set and a test set. Quite often, we first split the test set from the rest, and then we split the "rest" into train ...
user avatar
2 votes
1 answer
71 views

Why does XGBoost with cross-validation perform worse on test holdout than unvalidated model?

I have an XGBoost model that I fit on some X data directly out of the box: ...
user avatar
1 vote
0 answers
40 views

Bootstrapped Latin Partition

I'm having trouble understanding the Bootstrapped Latin partition method (as presented in Statistical validation of classification and calibration models using bootstrapped Latin partitions and ...
user avatar
  • 11
0 votes
0 answers
6 views

Do I need to separate unique IDs when doing out-of-time validation?

I'm doing a credit risk model and I thought that would be a good idea to validate out-of-time in multiple time periods (merging them to a big unique validation set). Each line of my dataset ...
user avatar
2 votes
2 answers
377 views

Logistic regression metric

I am interested to understand in which scenarios person should use sensitivity, specificity, and when should person opt for precision recall. On a high level I understand for a balanced data set we ...
user avatar
1 vote
1 answer
28 views

Generalization of model performance (AUC) and tuning of a catboost classifier

I was wondering if it is good practice to overfit on the training data while tuning a catboost classifier for a binary outcome. Wouldn't it be better to reguralize until validation error equals ...
user avatar
4 votes
1 answer
120 views

Why custom evaluation/scoring metric is causing overfitting in (cross) validation?

I am using machine learning to approach a balanced binary classification task. Some of rows are more important/valuable than others, so getting them right is extra important. Therefore, to accommodate ...
user avatar
1 vote
0 answers
26 views

Why does one model overfits faster than another?

Lets' say we have 2 models and we train both of them on the same data for the same number of epochs. When we monitor the validation loss, we realize that the point where the validation loss stops ...
user avatar
  • 143
1 vote
0 answers
19 views

Purpose of validation set in model.fit?

What is the purpose of the validation set in model.fit(…)? Does this touch the concept of cross validation at all? Afaik cross validation is done in an "...
user avatar
2 votes
0 answers
36 views

Final predictive model: using all available data?

Objective: Build a screening tool to identify people at risk of X. Approach: Using data from contexts A and B, we explored logistic regression models to predict X. We did forward & backward ...
user avatar
1 vote
0 answers
41 views

Including the validation set in the training set

I observed this in a Kaggle M5 competition notebook (cell 7, some explanation in cell 4) that inspired the competition winner, who used the same methodology to create "fake" validation data. ...
user avatar
  • 23
2 votes
0 answers
28 views

What is the proper way to externally validate clusters when I have only a sample of the dataset labeled, but want to cluster the entire dataset?

I have a dataset of text-based documents that I want to cluster. For a sample of this dataset (~10%) I have manually annotated labels (i.e., the ground truth). I would like to cluster this dataset to &...
user avatar
  • 21
0 votes
0 answers
14 views

Verifying if an already tuned model is overfitting or underfitting

Image that you are given a regression model $f_\theta(x)$ where the model parameters and have already been tuned and trained. How do you assess if this model is overfitting or underfitting ? You are ...
user avatar
1 vote
0 answers
75 views

How does Walk-Forward work with LSTM

I have been looking at how to split my data for training/validation/test for a timeseries using LSTM and have some conflicting thoughts I would like to get a bit more clarity on. I came across: QA1 ...
user avatar
  • 41
0 votes
1 answer
94 views

Data input: Expanding or Sliding Windows for LSTMs?

External research R1 (Stock Prediction with ML: Walk-forward Modeling by Chad Gray on 18/07/2018 at alphascientist.com) led me to believe that a sliding window is more favourable than an expanding ...
user avatar
  • 41
3 votes
1 answer
573 views

How to cross-validate a time series LSTM model?

I have been looking at how to split my data for training/validation/test for a timeseries using LSTM and came across: QA1 and QA2 Given I should implement walk-forward splits my depiction of it is: ...
user avatar
  • 41
1 vote
0 answers
14 views

What is possibly wrong in this Validation Accuracy and Training Accuracy [duplicate]

As general perception over training and validation accuracy is that if training accuracy is high and validation accuracy is marginally low, then it is most probably over fitting. Consider a case of ...
user avatar
2 votes
1 answer
28 views

Calculate predicted values after validation of logistic model

I have a simple logistic model, and I internally validated it using rms::validate in R. The estimated overoptimism for the intercept is -0.015 and for the slope is ...
user avatar
  • 123
0 votes
0 answers
78 views

How to correctly diagnose overfitting using all information: training set, validation set and test set?

I understand that overfitting is typically defined/described as the relationship between training set error and test set error - that overfitting is when a model performs significantly worse out-of-...
user avatar
0 votes
0 answers
11 views

Is there a way to measure or find the optimal accuracy of an association rule model which is to know if it is overfit or not

I'm currently new at data mining and have a problem understanding that most model such as classification and regression techniques are mostly using test and validation sets, which they can also apply ...
user avatar
  • 1
0 votes
0 answers
13 views

Which validation method should I use (lagged independant variables)?

I am a bit stuck on what validation method to use. I am working on a regression model using random forest. The model is trained on a dataset containing lagged independent variables (sensor data) and ...
user avatar
0 votes
0 answers
13 views

What other than overfitting can cause validation error to be significantly higher than training error [duplicate]

I am running a simple experiment. For simplicity, say I have 100k pictures of cats, with ground truth segmentation of their ears, and the network needs to correctly predict the segmentation. I split ...
user avatar
  • 101
0 votes
0 answers
37 views

No overlap in predictor for training and test dataset in random forest

I'm using a random forest classifier on a small dataset (a few hundred observations only). I randomly split the dataset into 80% training data and 20% held out testing data. When I tried to test, I ...
user avatar
  • 301
0 votes
0 answers
39 views

Validation of ml-model with highly imbalanced labels

I'm currently working on a project which is facing highly imbalanced class labels. So let's say we have 1000 good labels and 5 bad labels. At the moment I'm not sure how long i have to collect data to ...
user avatar
1 vote
1 answer
28 views

Psychometric test validation question

We are trying to validate a test for young children using IRT analyses (rasch/2pl). Models do not converge due to a floor effect. Teachers also took this assessment. Would it skew results if i ...
user avatar
  • 11
0 votes
0 answers
31 views

Test for a Bernoulli model with a single trial

I have a model that predicts Bernoulli probability parameters $p_i$ at $i\in[1..100]$ sites. To test this model, i can only take one trial at each one of the sites, resulting in $\approx10$ successes....
user avatar
  • 101

1
2 3 4 5
17