Questions tagged [validation]
The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.
867
questions
0
votes
0
answers
12
views
ROC curve analysis for when having a training/validation/test split
I have a dataset I split in training/validation/testing data for a binary classification model. The data is used as following:
Training data: for training the model (model weights, etc.)
Validation ...
- 115
0
votes
0
answers
17
views
Nan validation loss in mixed precision training of ResNet 50 [closed]
I am working in Tensorflow 2.8.0 with a ResNetv250 model as an image classificator. I have used a pretrained model with the ImageNet Weights as a feature extraction. It works like a charm. Now I want ...
- 51
0
votes
0
answers
6
views
Using validation data in optimization scenarios e.g. with genetic algorithms
I'm not too familiar with optimization algorithms e.g. genetic algorithms and I'm wondering whether it makes sense to employ a validation set in this context similarly to when we train a supervised ...
- 557
0
votes
0
answers
17
views
How to deal with overdispersion with glmmTMB for generalized linear models
I'll try to make it as brief as possible.
I'm trying to fit a glm to echolocation clicks count data using the glmmTMB function. I started with a Poisson glm and ...
0
votes
0
answers
10
views
Prognosis modelling data set
I want to externally validate and update a previously developed and published model (Model A) in my dataset. In the case of poor performance, I would like to develop a new model for the same outcome (...
- 23
1
vote
0
answers
43
views
Unusual results from XGBoost learning curves
I'm working on training an XGBoost classification model on time series data. Currently, I have a lot of data and it is hard to fit it all in memory, so I am trying to better understand if more data ...
- 21
0
votes
0
answers
22
views
Cluster Validation on Hamming Distances/K-Modes
I wondered if anyone could recommend internal cluster validation resources in R capable of estimating $k^*$ prior to using k-modes?
Therefore, I wondered if anyone knew of existing internal validation ...
- 244
1
vote
1
answer
68
views
Model evaluation metrics for comparing predicted probability accuracy across different datasets?
I'm working on an online model scoring framework, my goal is to be able to understand if my model's predictive performance is degrading week-over-week. I have a classification model (trained on binary ...
- 21
0
votes
1
answer
31
views
Do we only use cross-validation for hyperparameter tuning?
I am still unsure why we must use cross-validation here to validate the model, or it may be unnecessary.
Is it correct to use like indicated below? or do we have to combine it with hyperparameter ...
- 13
0
votes
0
answers
55
views
Rolling window validation for time series classification: good idea?
I have a time series dataset (interval = 10 minutes) that contains a user's visited locations. I derive several features from the timestamps to capture the user's trend: hour of the day, day of the ...
- 103
0
votes
0
answers
9
views
Normalizing data when retraining on the train and validation set
It is often good practice to normalize training data for numerical stability and faster convergence. When using a train-validation-test split, it is recommended to calculate normalization parameters (...
- 11
0
votes
0
answers
6
views
Validation of a stratified randomization plan
In my team, we are conducting clinical trials using stratified block randomization with random block size.
Depending on the user, the randomization list will be generated using custom SAS macros or R ...
- 1,400
0
votes
0
answers
12
views
Validation and testing by group with different distribution
I am working with medical data collected from different patients (10 different groups). Because of the nature of the imaging modality and different hardware parameter, the data distribution between ...
- 101
1
vote
1
answer
16
views
If not chosen all the data in the train partition, is it still k-fold cross validation?
I have a dataset of 900 images, distributed across 6 classes, with 150 images per class. To develop a classifier and assess its performance, I will utilize k-fold cross-validation. In this case, I ...
- 73
2
votes
1
answer
57
views
How to evaluate a Logistic Regression?
There are some previous post treating how to validate a logistic regression: Source 1 and Source 2.
But, still, those threads does not answer my question.
Therefore:
If a logistic regression predict ...
- 175
0
votes
0
answers
31
views
mlr3 : Benchmark different features selection methods
I have a simple question concerning my methodology.
I'm building some algorithms of machine learning to predict a binary outcome, using mlr3.
I optimized my different learners (svm, ranger, glmnet, ...
- 3
5
votes
2
answers
342
views
When dealing with data imbalance, shouldn't we never compare models based on validation loss, or at least weight it?
I know that when validating we are interested in knowing how the model performs in real-world scenarios, so we want the class ratios during validation/test to be the original ones.
Say, however, that ...
- 172
1
vote
1
answer
24
views
validation for time series models
Suppose that I have a model $f$ that is trained to predict hourly sales of 100 stores. Each day I retrain the model, and I have 2400 data points each day. Naively, I can split the data into sth like ...
- 155
0
votes
0
answers
29
views
External validation of diagnostic model: Performance in subgroups
We are performing an independent external validation of a 2 published diagnostic risk prediction tools (logistic regression models) which estimate the risk for having a specific disease (model A and ...
- 153
0
votes
0
answers
14
views
multiple datasets train-val-test split for time series
Suppose that I have data with dimension $(N,H,F)$, where $N$ represents the number of different datasets, $H$ is the history size and $F$ is the input size. how would you split it into a train-...
- 125
1
vote
0
answers
26
views
On the reliability of validation loss as a metric
I have the following plot of the training and validation loss from a deep neural network. The signature U-curve of the validation loss can be noticed. I want to use the validation loss as the metric ...
- 601
0
votes
0
answers
47
views
Proper conclusions from learning curves
I have a machine learning problem that I solve via nonlinear regression. I have 80 samples totally and try to understand is it useful to gather more data. For this, I plot learning curves in the ...
0
votes
1
answer
47
views
Bootstrap validation for panel data machine learning
A traditional machine learning validation strategy is to train on some data and check performance on some holdout data. When data are time-dependent, an obvious way to proceed is to train on early ...
- 55.1k
1
vote
0
answers
131
views
Is training on test set features (not labels) ok?
(Note: I'm using the word "feature" to refer to the content of the data for some test observation $i$: $x_i$. So "features" refers to many of these rows, e.g., $x_1, x_2$. I'm not ...
- 1,230
1
vote
1
answer
509
views
Is there a risk of overfitting when hyperparameter tuning a model
Is there a risk of overfitting when hyperparameter tuning a model using Optuna (or another hyperparameter tuning method ), with evaluation on a validation set and a large number of trials?
While a ...
- 105
0
votes
1
answer
88
views
Is repeated hyperparameter tuning can lead to overfitting?
I'm performing hyperparameter tuning for a classifier. After I finish, I'm updating the hyperparameter search space and re-tuning the hyperparameters again. I repeat this process a few times. In ...
- 105
1
vote
1
answer
182
views
Drawbacks of increasing K in k-fold Cross-Validation
I would like to know if there is any drawback to increasing K in K-fold CrossValidation, except the computational one.
- 301
0
votes
0
answers
167
views
AUC values of training and cross-validation are lower than AUC values of test set
I am training a Full model (logistic regression) and a few different models (LASSO, Elastic net, CART, random forest) to predict a certain clinical outcome.
I split my data into training and test sets ...
- 71
1
vote
1
answer
53
views
When can I ignore endogeneity problem?
Technically, endogeneity occurs when a predictor variable (x) in a
regression model is correlated with the error term (e) in the model.
This can occur under a variety of conditions, but two cases are
...
- 205
0
votes
0
answers
11
views
LSTM validation accuracy fluctuating [duplicate]
I am using LSTM to model time series data. My target variable is categorical so I am using one-hot encoding. The goal is to predict the target class based on the given time. My dataset spans over ...
0
votes
0
answers
19
views
If it is a good graph to show the relationship between the training error and test error after using optim function?
The following is code for optimization, during which I calculate the validation error(or test error) too.But I am not sure whether it is a good graph or if there is some problems with my code.Please ...
- 1
0
votes
0
answers
15
views
Is confirmatory factor analysis a way to validate a scale?
I am doing a translation and validation of a scale. I do not have any other measures, so I cannot do convergent/discriminant validity assessments. Is CFA enough to validate a scale, and if not what ...
0
votes
1
answer
36
views
How to compare different forecasting models (TS, ML, DL) using the same and replicable validation method [closed]
I am working on a research problem in which I compare an ARIMA, a lightGBM and an LSTM model across different hierarchical levels in a sales forecast setting (similar to the M5 forecasting competition ...
- 1
1
vote
0
answers
28
views
Do I need statistical test with nested-cv?
I'm working with a small data set and with 4 algorithms. The optimization process showed to be Very important as long as it improves a lot their performances. So, I'm using 10x5 nested-cv to estimate ...
0
votes
0
answers
16
views
AUC ROC validity merge experiments
Following situation:
I want to discuss my results. I repeated an experiment for 3 times for a binary classifier for validity. Now I want to draw a ROC-AUC curve.
What I do not want to do: I do not ...
- 33
0
votes
0
answers
15
views
Cluster Validation on Low-Dimensional Low-Separation Categorical Data
I've recently been using routines (k-modes, k-medoids, ROCK etc.) for clustering categorical survey data.
However, my data is low-dimensional and exhibits relatively low amounts of separability (i.e. ...
- 244
1
vote
1
answer
75
views
How naive forecast is used on validation and test sets?
I need to understand how naive method works when it comes to validation and test sets. I created the following time series and the data partition is as follows: the first 20 days as train, the next 5 ...
- 281
0
votes
0
answers
50
views
Training loss improving but predictions getting worse when using initialization with parameters close to "optimal"
I have a very simple toy problem that is part of a greater research project. I am trying to prove that a single hidden-layer MLP can learn $f(x) = x^2$. We all know a shallow MLP can do this. The only ...
- 101
0
votes
0
answers
16
views
Positive predictive value of test only dependent on prevalence and FPR?
I have a question regarding the PPV of a test in the context of the famous conclusion of Bayes test in tests for rare diseases:
The gist is that even highly sensitive and specific tests will make a ...
- 307
1
vote
1
answer
43
views
Do transformations caricaturize the data? Are corresponding outcomes of hypothesis tests artefacts?
I am getting different significance outcomes when using non-parametric tests on the raw data versus log-transforming the data and then applying parametric tests. Which one is more valid?
Please ...
- 111
0
votes
0
answers
24
views
Correct presentation of model develoment with post-estimation shrinkage after internal validation in R
I'm developing a prediction model (binary outcome) in R and I have the following question:
After fitting my model with fit.mult.impute from the hmisc package (i dealt with some missing values in my ...
0
votes
0
answers
92
views
How to use early stopping with imbalanced data?
I'm wondering if it is theoretically sound to use a metric such as binary F1-score as part of my early stopping criteria in tuning a model during training. I'm working with a collection of binary ...
- 121
0
votes
0
answers
17
views
Validation of Data on SPSS [duplicate]
I have the data of 150 participants [2 different methods that assess the same thing (Blood pressure), one of which is considered as "gold standard method"] and I want to validate them on ...
user376879
0
votes
0
answers
13
views
Which non parametric statistical test to choose for Ground Truth validation?
I am preparing a dataset of source code vulnerabilities of GitHub projects. In the dataset, I have isVulnerabilityPresent column which has value either True or False meaning that vulnerability is ...
- 101
0
votes
0
answers
56
views
When can we consider a neural network overfit?
The concept of overfitting is widely talked about but opinions on the determining factor for whether a network is starting to overfit seems to differ widely. I guess this is broadly combined with: &...
0
votes
0
answers
37
views
How to find the validation R² and RMSE of a caret random forest model with out-of-bag validation?
I trained six random forest models using the train function from the caret R package using out-of-bag valiadation (...
0
votes
0
answers
23
views
Cross validation on bootstrap data
I am performing a dirichlet model for different species using a small sample size (between 8 to 20 samples per each).
Since my dataset is small, I bootstrap my data with 1000 iterations, averaging 3 ...
- 377
1
vote
0
answers
53
views
Clarification for the test and validation set
I have been trying to figure out the difference between validation and test set used in the machine learning studies. In almost all places, I found that I have to split the dataset into three parts (...
- 33
0
votes
0
answers
13
views
How Do You Generate a Score for a Risk Assessment Variable Using Weights Derived from Logit Coefficients?
I am validating an 8-item risk assessment tool designed to classify inmates to custody levels and predict institutional misconduct within prisons (0 = No misconduct; 1 = Misconduct). I would like to ...
1
vote
1
answer
20
views
Why the validation error does not decrease with a training of 30 epochs but decreases with a training of 60 epochs
When i train my model with 30 epochs, the training and validation error curves seems to stagnate:
However, when i train my model with 60 epochs, the training and validation error curves start to ...
- 11