Skip to main content

Questions tagged [cross-validation]

Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

In X-learner uplift modeling, predictions from the 1st-stage models help train the 2nd-stage models. What data splits should these predictions be on?

In uplift modeling with an X-learner metalearner (Künzel et al. 2019), predictions from the two first-stage models are used in training the two second-stage models. Question: What datasets/splits ...
naive_bayesian's user avatar
0 votes
0 answers
28 views

How to determine lambda for graphical lasso?

I am trying to figure out how to determine lambda for a graphical lasso. I have found that someone had the exact same question that me 9 years ago. I was wondering if anything exists in R to determine ...
Simon's user avatar
  • 1
1 vote
0 answers
11 views

Combine back- and forecast errors for cross-validation

Suppose I have a procedure to predict the timeseries value $Y_{t+k}$, where $t$ is the current period and $k \geq 1, 2, \dots$. Now, I want to estimate the procedure's out-of-sample performance. The ...
bodhi's user avatar
  • 21
1 vote
3 answers
55 views

Can I apply data augmentation to the test set?

I'm working with a dataset of 102 rows (tabular data), from which I'm using 91 for training and 11 for testing. I'm using data augmentantion through the addition of gaussian noise for the training set....
Vinicius Maia's user avatar
0 votes
0 answers
36 views

Environmental filtering versus spatial resampling in species distribution modeling

I am building species distribution models using machine learning models based on GBIF data (presence-only data) and working on a very large spatial scale, encompassing all of North America. Before ...
Marine Régis's user avatar
2 votes
0 answers
23 views

Using whole training set for choosing model

I am working on a classification problem with what I understand as a big dataset. I have first of all splitted it in my "train" dataset and the "test" one. (Actually I am convinced ...
Videgain's user avatar
0 votes
0 answers
27 views

Comparing Models with Unequal Sample Sizes

I have performed an association analysis where I have associatiated several different perdictor variables to a dependent variable. For each predictor, I run two models and compare them via the ...
CAM_etal's user avatar
2 votes
1 answer
35 views

Best Practices for Splitting Data in a Repeated Measures Classification Problem

I am working on a classification problem involving repeated measures. My objective is to classify positive patients as early as possible. In my practical application scenario, once the target becomes ...
jpsca1293's user avatar
0 votes
0 answers
20 views

Kfold cross val in Regression model

How to use K-fold CV to evaluate my regression model performance to calculate the R2, MAE and MSE in the train set to make the model more robust? This code below refers to the tuned model and I'm ...
Vinicius Maia's user avatar
1 vote
0 answers
37 views

how to approximate the eigendecomposition of a correlation matrix when the data have been standardized?

Context I am working to develop a penalized regression framework that will scale up to analyzing high dimensional data with a certain correlation structure. Let $X$ represent an $n \times p$ matrix of ...
Tabitha Peter's user avatar
1 vote
1 answer
28 views

Time series cross validation for trend + tree model

I have a time series data set with ~3 years of data sampled daily from 2021 to 2024. The data set exhibits a trend, and clear cycles with periods of 1 year and 1 week. My goal is to forecast ~3 ...
anon12345's user avatar
0 votes
0 answers
45 views

Help with completing a derivation of usefulness of cross-validation

This question is raised as a result of my attempt to answer this other question of mine. Let's refer to all our prior knowledge, both explicit and implicit, as $X_\text{true}$. Almost always, we are ...
Feri's user avatar
  • 197
0 votes
0 answers
22 views

Statistical Significance Testing for Nested Cross-Validation in ML Experiment

I am currently working on an ML experiment where I use a nested 5-cross validation procedure and obtain a NDCG@10 scores for each test user. I am comparing 6 different ML algorithms and have data for ...
Bernhard's user avatar
  • 101
0 votes
0 answers
14 views

How to split data when training and tuning the meta learner in stacking?

I have a simple yet tricky conceptual question about the data splitting of a meta learning process. Assume I have a simple X_train, ...
Yann's user avatar
  • 43
0 votes
0 answers
20 views

k-fold cross-validation for extremely imbalanced classes?

I'm doing a classification project with two imbalanced classes. I'm aware that for k-fold cross-validation in Python one can use the option "stratify" when making the splits to account for ...
leyjfk6's user avatar
3 votes
1 answer
33 views

Metrics of a classification problem after k-fold cross-validation

I have a classification problem in which the dataset is made up of several smaller datasets $D_1, \cdots, D_k$. I want to learn a classifier $h_i$ for each $i=1,\cdots, k$, using $D_1, \cdots, D_{i-1},...
user605734 MBS's user avatar
0 votes
0 answers
13 views

Comparing GLMs with different fitted distributions

I have a scenario where I need to compare some generalized liner models (with same link function, target variable, but not necessarily nested) with k fold cross validation, using a cost function to ...
user101874's user avatar
0 votes
0 answers
13 views

How does highly imbalanced test data in certain splits of k-fold time-series cross-validation affect model performance?

I am working on a time-series classification (TSC) problem using k-fold time-series cross-validation (TSCV) to evaluate the performance of my models. My training data for each split is fairly balanced,...
Tov Nephesh's user avatar
1 vote
0 answers
30 views

Significant performance drop between train and validation set

I am trying both Lgbm and RandomForest for a classification, and I observe the same problem. I am using various metaparams to prevent overfitting, such as max_depth, num_trees (keeping it small for ...
Baron Yugovich's user avatar
15 votes
4 answers
1k views

Train-validation-test split for small and unbalanced dataset?

I have a dataset of around 100 rows, each with around 400 features. 93 of them are class 0, and 7 are class 1. I want to be able to split my 100 examples into a train set, a validation set, and a test ...
Thao Nguyen's user avatar
0 votes
0 answers
14 views

GridSearchCV performs worse than baseline

I'm working on a binary classification problem using scikit-learn. One of the models I've tested is KNeighborsClassifier, for ...
AndreaTerenz's user avatar
2 votes
1 answer
32 views

Proper usage of K-fold cross validation and finalizing model

I am trying to learn about k-fold cross validation. I am using it on Kaggle dataset of brain tumors MRI trying to classify the images. Kaggle provides two directories Training with 5712 images and ...
Daniel11's user avatar
2 votes
1 answer
74 views

Is my understanding/approach to nested cross-validation, final model tuning correct?

I am training a SVM on limited training data with unbalanced classes. Here are the things that I want to do: 1.) I want to make a statement of the generalizability ...
curious's user avatar
  • 115
2 votes
1 answer
32 views

How to interpret the results of a classifier when train/test method gives much better results than cross validated one?

I need your help to understand a situation where using train and test set produces perfect results (in terms of accuracy, precision, and recall) but when cross validation is used, the accuracy on ...
letdatado's user avatar
  • 367
0 votes
0 answers
46 views

Multivariate ljung-box test: how to pick the degree of freedom for the chi-squared distribution?

I have a model with $p$ inputs and $q$ outputs. I want to check the autocorrelation of the output residuals, defined as $\epsilon(t) = y(t) - \tilde y(t)$, being $y$ the model output and $\tilde y$ ...
Barzi2001's user avatar
  • 113
2 votes
0 answers
47 views

Leave-one-*in* cross validation

I'm trying to argue that my model is very robust. Given are $N$-many x-y pairs as samples. Iterating over all samples, for each sample a model is trained using only this single sample. This model is ...
mafu's user avatar
  • 143
0 votes
0 answers
10 views

Interpretation Comparison Bayesian Models with Leave-One-Out Cross Validation

I'm currently working on comparing two Bayesian linear models using the brms package in R, with a dataset of 400 participants. The models differ in that one ...
JKas's user avatar
  • 21
10 votes
1 answer
400 views

Bayesian Justification of Cross-validation

If I understand correctly, K-fold cross-validation is supposed to approximate expected log predictive density (ELPD), which is defined as $\mathop{\mathbb{E}}_{D_{new}\sim P(.|M_{true})}\log P(D_{new}|...
Feri's user avatar
  • 197
0 votes
0 answers
30 views

Can you store the value of the predicted variable (Y) at each fold and then correlate the predicted values with the actual data?

In particular, imagine to have a set of features (X) that I use to predict a continuos variable (Y). Is it possible to use elastic net, in a cross-validation framework, use it to predict the value of ...
sup_use's user avatar
2 votes
0 answers
22 views

Is averaging RMSE values across cross-validation folds mathematically invalid?

Correct me if I'm wrong, but it seems that both scikit-learn and tidymodels will average the metrics of choice (RMSE, R2, etc.) ...
Arthur's user avatar
  • 566
0 votes
0 answers
17 views

How to Train a Model on the Whole Outer-loop Training Set in Nested Cross-validation?

I'm implementing nested cross-validation for a machine learning project and need some clarity on the training process using the outer-loop training set. Here’s a summary of my process: Outer-loop ...
Surayuth Pintawong's user avatar
0 votes
1 answer
47 views

What can I do about model tuning parameter instability?

I am trying to determine the importance of watershed characteristics on the slope of the concentration-discharge relationship for several rivers. I am using partial least squares regression (plsr) ...
Ryan's user avatar
  • 1
0 votes
0 answers
11 views

Can we use decomposed time series in a test set to obtain models accuracy metrics?

Currently I'm cross validating (rolling time window, different test lengths) several forecast models to obtain performance comparison on highly skewed time series (due to true ocassional outliers). My ...
Tom's user avatar
  • 491
1 vote
0 answers
90 views

Hyperparameter tuning for small datasets

I have about 10 small imbalanced datasets (some of them only have about 150 samples). I want to try a bunch of balancing techniques on some models. For that, I'm using the repeated stratified cross-...
beautifularmy's user avatar
0 votes
0 answers
12 views

Is GroupKFold needed if some samples have some of their feature values equal?

I am given a dataset $D$ of 10k enzyme-substrate complexes having a lock-key relationship, with each sample (complex) being characterized by enzyme features $x_e$ and substrate features $x_s$. That is,...
ado sar's user avatar
  • 477
0 votes
0 answers
17 views

Regression metrics calculation for each fold vs calculation at the end of k-fold cross validation [duplicate]

I stumbled upon a small realization while I was calculating fit metrics during k-fold cross-validation. Please refer to the following images: The approach of calculating R² or other fit metrics for ...
Tino D's user avatar
  • 268
0 votes
0 answers
11 views

Select classification model using nested cv and bootstrap auc confidence interval

My goal is to find the best 1 model out of 55 classification models. I first ran nested cv on 55 models to see which model had better generalization. The AUC score was used as an evaluation indicator. ...
JAE's user avatar
  • 89
1 vote
1 answer
35 views

Accuracy "overfits" but loss doesn't?

I'm perplexed as to why my loss doesn't go up when the accuracy goes down (after about 40 epochs). Isn't it possible to tell overfitting from the loss curve alone? (I'm of course referring the ...
Tfovid's user avatar
  • 795
0 votes
0 answers
31 views

How can I assess internal validation (discrimination, calibration) of a Fine-and-Gray competing risk model, fitted using a MFP-algoritm in Stata?

I'm a post-doc at Karolinska Institute, and I'm working on developing a Fine-and-Gray competing risk model to predict endometrial cancer recurrence/progression with death as a competing event. I have ...
Rasmus Green's user avatar
1 vote
0 answers
13 views

Consistent way of doing paired-trial validation (and leave-one-dataset-out validation)

In paired-trial validation, a statistical (ML) models are trained on $n$ datasets separately and then applied to other datasets, as a way of estimating the generalization of the models obtained. ...
Roger V.'s user avatar
  • 4,429
4 votes
1 answer
74 views

repeated measures mixed effects model with time-varying covariates in r

Assuming that we have longitudinal data on pulmonary fibrosis with some patients undergoing transplant while others received medical treatment. Each patient is represented by many rows depending on ...
Mohamed Rahouma's user avatar
0 votes
0 answers
72 views

Is it necessary to average the shap values ​that have been processed through cross validation?

Is it necessary to average the shap values ​​that have been processed through cross validation? I saw an online site called 'towardsdatascience' that calculated the shap value through 10 CV and then ...
JAE's user avatar
  • 89
2 votes
1 answer
27 views

Why is there a difference in approach between cross-validation and using the test set in ISL's explanation of survival analysis?

The following is from Introduction to Statistical Learning Python edition, page 485, from the survival analysis chapter. They make a distinction between the cross-validation process (to choose a ...
NovicePatience's user avatar
0 votes
0 answers
33 views

I screwed-up model selection but ended-up with a very good model; am I ok?

In a recent experiment, I made an oversight: I divided my data into training and testing sets and conducted cross-validation for model selection and hyperparameter tuning after having applied Boruta (...
Alek Fröhlich's user avatar
4 votes
2 answers
64 views

How to approach dataset splitting for building time-series models?

Suppose I have 100 observations of time series data $x_1,...,x_{100}$, and that I want to split the data into a train set, a validation set, and a test set. I know that the train set must have smaller ...
David's user avatar
  • 1,276
0 votes
0 answers
9 views

Selection of important features through cross validation and shape value importance

To extract important features for the binary classification problem, recursive feature elimniation was performed based on the importance value of the shap value through nested cv. The first thing I am ...
JAE's user avatar
  • 89
0 votes
1 answer
36 views

Random forest cross-validaton by patient

I have a dataset of various features from 10 patients and 10 controls. Each patient has many data points. Random forest does an amazing job in predicting whether a data point is from a patient or a ...
SuperDuperMario's user avatar
1 vote
1 answer
62 views

How do you train-test split an imbalanced dataset?

I have an imbalanced dataset and I'm trying to predict a binary target. The minority class amounts to approximately 0.4% of all observations (60 million observations from which 250K belong to the ...
Arturo Sbr's user avatar
6 votes
2 answers
120 views

Comparison of roc-auc values ​through cross-validation for feature selection

Through 10 cv, the roc-auc value was obtained as follows. At first, I tried to select the feature with the highest average roc-auc value, but I had doubts about whether the difference in these scores ...
JAE's user avatar
  • 89
0 votes
0 answers
11 views

How to perform proper cross-validation in case of a regime shift?

Assume that K-Fold cross validation is used on time series and the data experiences a regime shift after the 1st fold (assume it is the leftmost one for simplicity). Regime shift in this context means ...
Kreol's user avatar
  • 121

1
2 3 4 5
70