Skip to main content

Questions tagged [validation]

The process of assessing whether the results of an analysis are likely to hold outside of the original research setting. DO NOT use this tag for discussing 'validity' of a measurement or instrument (such as that it measures what it purports to), use [validity] tag instead.

Filter by
Sorted by
Tagged with
4 votes
1 answer
29 views

Validating binary prediction model

Suppose we have a model that predicts for binary event $e$ ($0$ or $1$) with a single output $p$ (the expected probability $e$ occurs). If we are able to compare $p$ with the true value of $e$ ($0$ or ...
shrizzy's user avatar
  • 141
4 votes
2 answers
283 views

Is domain knowledge external validation in clustering?

I have cluster results with good values on etc Silhuette Width. The cluster sizes are: 4998, 1, 1 which isn't good knowing my customers doesn't have that particular partition (it's more balanced). I ...
ExchangedVisual111's user avatar
1 vote
1 answer
26 views

Are there strategies for measuring accuracy of Euclidean distance-based similarity without ground truthing?

I have subjects with about 200 features each. These feature vectors are stored in a vector database, where similarity searching with Euclidean distance is used to find subjects that are similar to a ...
T_d's user avatar
  • 13
0 votes
0 answers
17 views

How were the asymmetric recovery ranges in Table A5 of Appendix F from AOAC determined?

I am trying to understand how the recovery ranges in Table A5 of Appendix F from AOAC (https://www.aoac.org/wp-content/uploads/2019/08/app_f.pdf) were determined. I did not understand how the ...
Éderson D'Martin Costa's user avatar
0 votes
0 answers
11 views

What to do with features causing data drift?

I dispose of labeled train data and unlabeled test data. I want to tune and validate a classifier on train data in such a way that it can have good performance on test. By conducting some ...
Yann's user avatar
  • 43
0 votes
0 answers
8 views

Validation accuracy dip and recovery when restarting training

i was fine-tuning this large language model with Stochastic Gradient Descent and mid epoch i stopped training, and saved the model weights. Then at a later time, reloaded the weights and restarted the ...
clam's user avatar
  • 348
0 votes
1 answer
43 views

Are the p-values obtained on the same sample using synthetic AA tests (Monte Carlo) independent values?

Let's say we have the following procedure. We take a fixed sample of size n and perform the procedure 1000 time: we divide (split) it equally into 2 groups; we calculate p value using the F function (...
Романов Андрей's user avatar
4 votes
2 answers
98 views

How to split and sample "Panel Data" when training a Logistic Regression to predict future outcomes

Introduction I have panel data where customer behavior is observed over time. For each customer at a given reference date, I have a lookback window of 12 months for generating features, and a look ...
Esben Eickhardt's user avatar
1 vote
0 answers
25 views

Evaluate hierarchical clustering with partial ground truth

I am performing hierarchical clustering, and I need to decide which agglomeration method to use. While I don't have a ground truth, I know that some datapoints should be closer together: for example, ...
Alexlok's user avatar
  • 143
1 vote
1 answer
20 views

Validation Accuracy higher if trained on training set than in validation set

So, I'm having a problem with the validation of a model, in particular, I'm trying a linear readoff (a logistic regression attached to a middle layer of a neural network) In particular, if I train the ...
Alberto's user avatar
  • 1,207
1 vote
0 answers
7 views

Image classification metrics

I have been working on an image classification task using CNNs and getting some puzzling results. My training, validation and test loss keep going down with epochs and are comparable. So this might ...
Nithin's user avatar
  • 11
0 votes
0 answers
17 views

Leaving duplicated entries in a dataset at pretraining stage

I'm adopting a fine-tuning approach after having pretrained a deep learning model (transformer) on a source dataset (let's call it dataset A) and then fine-tuning it on a target dataset (B). Dataset A ...
James Arten's user avatar
0 votes
0 answers
25 views

How can I assess internal validation (discrimination, calibration) of a Fine-and-Gray competing risk model, fitted using a MFP-algoritm in Stata?

I'm a post-doc at Karolinska Institute, and I'm working on developing a Fine-and-Gray competing risk model to predict endometrial cancer recurrence/progression with death as a competing event. I have ...
Rasmus Green's user avatar
1 vote
0 answers
13 views

Consistent way of doing paired-trial validation (and leave-one-dataset-out validation)

In paired-trial validation, a statistical (ML) models are trained on $n$ datasets separately and then applied to other datasets, as a way of estimating the generalization of the models obtained. ...
Roger V.'s user avatar
  • 4,421
0 votes
0 answers
11 views

External Validation of Prediction Models - Cox vs Logistic Regression, pros and cons?

I have made a Cox PH model for prognostication of a favourable outcome (with competing events being death). I now have an independent set of data to do external validation on this model. The primary ...
Hong's user avatar
  • 13
1 vote
1 answer
100 views

Interpreting a rootogram

I recently needed to use a Zero-inflated Negative Binomial and Hurdle Negative Binomial models to model some data. When finding ways to assess the goodness of fit, I came across rootograms. But I am ...
Bileobio's user avatar
0 votes
1 answer
25 views

Which data subset should be used for interpretable machine learning (IML)? [duplicate]

In a machine learning workflow, we need to split the dataset into training and test sets. We train several candidate models (typically tuned with hyperparameter optimization) on the training set and ...
Tripartio's user avatar
  • 2,206
2 votes
1 answer
54 views

What are the implications on prediction if a regression model with heteoscedastic errors is considered for analysis?

I have a simple linear regression model, and I tried validating if the model fit for purposes. One of the tests is Bruesch Pagan test on the residuals but the result of the test shows that the error ...
Abdul Mateen Hashim's user avatar
0 votes
0 answers
55 views

Evaluating the statistical significance of survey's responses

I'm a teacher doing some basic research but I'm out of my waters in anything statistical. Recently I submitted a paper based on a student survey which aimed at evaluating the impact of a new ...
MyName's user avatar
  • 101
1 vote
0 answers
41 views

How to statistically validate samplers? Is Chi-Square test relevant for synthetic data?

I am developing sampling procedures for different kinds of discrete objects (rankings, sets, etc.). In most cases I know the theoretical distribution of the objects I'm sampling. My idea is thus to ...
Siolan's user avatar
  • 11
0 votes
0 answers
16 views

The accuracy of my validation set is always the same

I am training a CNN model which is used for a multi-label classification task. My training data set has 5000 data points, every data point is a 100000 long 1-D array. So the shape of my training set ...
user avatar
1 vote
0 answers
39 views

Sample size calculation for external validation of prediction model [closed]

I am confused about calculating degrees of freedom for external validation of a clinical prediction model. The calculation for sample size is as follows: We will include 4,875 patients. The ...
Komal Valliani's user avatar
0 votes
0 answers
34 views

Reporting internal validation from .632 bootstrap in caret

I am developing Machine learning prediction models using the caret package in r (Elastic net, SVM, Random Forest, XGBoost). I have 650 cases with 104 having the event of interest. Instead of splitting ...
Dwayne T's user avatar
0 votes
0 answers
11 views

Do you use the final validation accuracy or best validation accuracy during training to fine-tune hyperparameters?

I am currently training a neural network on the EMNIST dataset for an assignment. We are asked to select the best combinations of hyperparameters (inclusion probability for the dropout layer and ...
Gregor Hartl Watters's user avatar
1 vote
1 answer
38 views

Refitting model on entire dataset to use in production to make prediction on latest data?

I have a question regarding calibration (train) and holdout (test) periods.I have time based data. I split my data into two sets. Fit/Train (first 2/3 of data) and validation/testing (latest third). ...
Birk's user avatar
  • 33
1 vote
0 answers
12 views

what does it mean to have slightly higher valid error rate, very high and similar valid error rate compared to train and test?

the dataset is split into 3 categories: train, test, valid. after training the error rates we see have these values. case 1 1% error on the training set. 5% error on training dev set. 5% error on the ...
ERJAN's user avatar
  • 111
0 votes
0 answers
16 views

How to test statistical improvement under model prediction uncertainty?

I have a regression model, predicting a popularity of a text. I have its performance metrics on test set, e.g. RMSE and MAE. This gives me an uncertainty estimate about its predictions. Now I want to ...
qalis's user avatar
  • 238
0 votes
0 answers
35 views

ROC curve analysis for when having a training/validation/test split

I have a dataset I split in training/validation/testing data for a binary classification model. The data is used as following: Training data: for training the model (model weights, etc.) Validation ...
Alb's user avatar
  • 115
0 votes
0 answers
6 views

Using validation data in optimization scenarios e.g. with genetic algorithms

I'm not too familiar with optimization algorithms e.g. genetic algorithms and I'm wondering whether it makes sense to employ a validation set in this context similarly to when we train a supervised ...
James Arten's user avatar
0 votes
0 answers
130 views

How to deal with overdispersion with glmmTMB for generalized linear models

I'll try to make it as brief as possible. I'm trying to fit a glm to echolocation clicks count data using the glmmTMB function. I started with a Poisson glm and ...
Carlos Benítez Collins's user avatar
0 votes
0 answers
11 views

Prognosis modelling data set

I want to externally validate and update a previously developed and published model (Model A) in my dataset. In the case of poor performance, I would like to develop a new model for the same outcome (...
imunicorn's user avatar
1 vote
0 answers
90 views

Unusual results from XGBoost learning curves

I'm working on training an XGBoost classification model on time series data. Currently, I have a lot of data and it is hard to fit it all in memory, so I am trying to better understand if more data ...
Ted's user avatar
  • 21
0 votes
0 answers
48 views

Cluster Validation on Hamming Distances/K-Modes

I wondered if anyone could recommend internal cluster validation resources in R capable of estimating $k^*$ prior to using k-modes? Therefore, I wondered if anyone knew of existing internal validation ...
EB3112's user avatar
  • 244
1 vote
1 answer
204 views

Model evaluation metrics for comparing predicted probability accuracy across different datasets?

I'm working on an online model scoring framework, my goal is to be able to understand if my model's predictive performance is degrading week-over-week. I have a classification model (trained on binary ...
Ted's user avatar
  • 21
0 votes
1 answer
83 views

Do we only use cross-validation for hyperparameter tuning?

I am still unsure why we must use cross-validation here to validate the model, or it may be unnecessary. Is it correct to use like indicated below? or do we have to combine it with hyperparameter ...
vdu16's user avatar
  • 13
0 votes
0 answers
72 views

Rolling window validation for time series classification: good idea?

I have a time series dataset (interval = 10 minutes) that contains a user's visited locations. I derive several features from the timestamps to capture the user's trend: hour of the day, day of the ...
sander's user avatar
  • 103
0 votes
0 answers
20 views

Normalizing data when retraining on the train and validation set

It is often good practice to normalize training data for numerical stability and faster convergence. When using a train-validation-test split, it is recommended to calculate normalization parameters (...
Vityou's user avatar
  • 11
0 votes
0 answers
12 views

Validation of a stratified randomization plan

In my team, we are conducting clinical trials using stratified block randomization with random block size. Depending on the user, the randomization list will be generated using custom SAS macros or R ...
Dan Chaltiel's user avatar
  • 1,440
1 vote
1 answer
19 views

If not chosen all the data in the train partition, is it still k-fold cross validation?

I have a dataset of 900 images, distributed across 6 classes, with 150 images per class. To develop a classifier and assess its performance, I will utilize k-fold cross-validation. In this case, I ...
noone's user avatar
  • 73
2 votes
1 answer
365 views

How to evaluate a Logistic Regression?

There are some previous post treating how to validate a logistic regression: Source 1 and Source 2. But, still, those threads does not answer my question. Therefore: If a logistic regression predict ...
Another.Chemist's user avatar
5 votes
2 answers
391 views

When dealing with data imbalance, shouldn't we never compare models based on validation loss, or at least weight it?

I know that when validating we are interested in knowing how the model performs in real-world scenarios, so we want the class ratios during validation/test to be the original ones. Say, however, that ...
raquelhortab's user avatar
2 votes
1 answer
54 views

validation for time series models

Suppose that I have a model $f$ that is trained to predict hourly sales of 100 stores. Each day I retrain the model, and I have 2400 data points each day. Naively, I can split the data into sth like ...
koch's user avatar
  • 195
1 vote
0 answers
42 views

On the reliability of validation loss as a metric

I have the following plot of the training and validation loss from a deep neural network. The signature U-curve of the validation loss can be noticed. I want to use the validation loss as the metric ...
wd violet's user avatar
  • 777
0 votes
1 answer
61 views

Bootstrap validation for panel data machine learning

A traditional machine learning validation strategy is to train on some data and check performance on some holdout data. When data are time-dependent, an obvious way to proceed is to train on early ...
Dave's user avatar
  • 64.2k
7 votes
2 answers
448 views

Is pretraining on test set texts (without labels) ok?

Edit: after skimming this paper6, I narrowed the scope of this question to NLP problems. Relevant excerpt from the abstract (emphasis my own): We demonstrate that unsupervised preprocessing can, in ...
chicxulub's user avatar
  • 1,420
4 votes
1 answer
1k views

Is there a risk of overfitting when hyperparameter tuning a model

Is there a risk of overfitting when hyperparameter tuning a model using Optuna (or another hyperparameter tuning method ), with evaluation on a validation set and a large number of trials? While a ...
Amit S's user avatar
  • 57
0 votes
1 answer
176 views

Is repeated hyperparameter tuning can lead to overfitting?

I'm performing hyperparameter tuning for a classifier. After I finish, I'm updating the hyperparameter search space and re-tuning the hyperparameters again. I repeat this process a few times. In ...
Amit S's user avatar
  • 57
2 votes
1 answer
758 views

Drawbacks of increasing K in k-fold Cross-Validation

I would like to know if there is any drawback to increasing K in K-fold CrossValidation, except the computational one.
Simone's user avatar
  • 321
1 vote
1 answer
117 views

When can I ignore endogeneity problem?

Technically, endogeneity occurs when a predictor variable (x) in a regression model is correlated with the error term (e) in the model. This can occur under a variety of conditions, but two cases are ...
Laiy's user avatar
  • 245
0 votes
0 answers
14 views

LSTM validation accuracy fluctuating [duplicate]

I am using LSTM to model time series data. My target variable is categorical so I am using one-hot encoding. The goal is to predict the target class based on the given time. My dataset spans over ...
user2585933's user avatar

1
2 3 4 5
18