Questions tagged [cross-validation]

Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.

3,497 questions
Filter by
Sorted by
Tagged with
29 views

Review of Methodology and RMSE, MAE for multi-class Classification of Air Quality Index Categories

I am working on my university thesis and have proposed the following methodology for predicting the Air Quality Index (AQI) and subsequently classifying it into categories. There are six AQI ...
1 vote
23 views

What is the performance of a "meta" learner that performs internally CV for model selection?

I am trying to understand the proof that reporting CV performance during model selection as performance estimate is optimistically biased. The steps in the proof are the following: Let $p_i, \pi_i$ ...
• 477
11 views

Setting data filters by optimizing on goodness-of-fit parameter [closed]

I am not a mathematician, statistician or data scientist, but someone trying to apply timeseries analysis to my scientific domain. So pardon any ignorance in my question :D The data and statistical ...
20 views

Based on the scatter-plot that i have made to validate my regression model, that could be considered a good model?

I have a gradient-boost regressor model, i'm predicting value of sentences, which often are round numbers (which makes it annoying to validate). I have created a scatter-plot to validate my model, x-...
26 views

LOOCV test error is different by mannually "for() loop" or cv.glm() [closed]

It's about Question 7 in Chapter 5 of book "An Introduction to Statistical Learning with Applications in R". Below is part of the question's text: In Sections 5.3.2 and 5.3.3, we saw that ...
• 1,039
13 views

Using different performance metrics in outer vs inner loop in nested cross-validation

What are the benefits or disadvantages of using 2 different performance metrics in inner versus outer loops of nested cross-validation? I want to use both AUC-ROC and log loss but can't decide if I ...
11 views

Can I use voting in nested cross validation to obtain estimator?

NEWBIE ALERT: First post, so be gentle :-) I have looked at nested cross validation (as e.g. in this Medium article). I understand that I am able to obtain averaged scores from the outer loop and that ...
16 views

Model performing poorly after cross validation

After using cross-validation to see how a custom predictive function performs on unseen data, I applied to function to the original dataset, and the performance (based on coefficient of determination) ...
1 vote
25 views

Standard Error of repeated nested cross-validation

Is there a model-agnostic formula for the standard error of K-Fold cross validation, nested cross-validation, or repetead nested cross-validation prediction results? I just stumbled about this post ...
6 views

Time Series Cross Validation by skicit learn and Cross Validation by Prophet

I have confusion between time series cross validation by skicit learn and cross validation by Prophet. So I'm trying to compare lstm algorithm with prophet and method The split data used for LSTM is ...
17 views

Reducing Variance with Regularization in LOOCV for Small Datasets

I have a small dataset and I am considering using Leave-One-Out Cross-Validation (LOOCV) to evaluate my model. I understand that cross-validation, in general, is a method to assess a model's ...
27 views

In X-learner uplift modeling, predictions from the 1st-stage models help train the 2nd-stage models. What data splits should these predictions be on?

In uplift modeling with an X-learner metalearner (Künzel et al. 2019), predictions from the two first-stage models are used in training the two second-stage models. Question: What datasets/splits ...
132 views

How to determine lambda for graphical lasso?

I am trying to figure out how to determine lambda for a graphical lasso. I have found that someone had the exact same question that me 9 years ago. I was wondering if anything exists in R to determine ...
• 31
1 vote
13 views

Combine back- and forecast errors for cross-validation

Suppose I have a procedure to predict the timeseries value $Y_{t+k}$, where $t$ is the current period and $k \geq 1, 2, \dots$. Now, I want to estimate the procedure's out-of-sample performance. The ...
• 21
1 vote
60 views

Can I apply data augmentation to the test set?

I'm working with a dataset of 102 rows (tabular data), from which I'm using 91 for training and 11 for testing. I'm using data augmentantion through the addition of gaussian noise for the training set....
36 views

Environmental filtering versus spatial resampling in species distribution modeling

I am building species distribution models using machine learning models based on GBIF data (presence-only data) and working on a very large spatial scale, encompassing all of North America. Before ...
25 views

Using whole training set for choosing model

I am working on a classification problem with what I understand as a big dataset. I have first of all splitted it in my "train" dataset and the "test" one. (Actually I am convinced ...
• 21
31 views

Comparing Models with Unequal Sample Sizes

I have performed an association analysis where I have associatiated several different perdictor variables to a dependent variable. For each predictor, I run two models and compare them via the ...
• 75
35 views

Best Practices for Splitting Data in a Repeated Measures Classification Problem

I am working on a classification problem involving repeated measures. My objective is to classify positive patients as early as possible. In my practical application scenario, once the target becomes ...
21 views

Kfold cross val in Regression model

How to use K-fold CV to evaluate my regression model performance to calculate the R2, MAE and MSE in the train set to make the model more robust? This code below refers to the tuned model and I'm ...
1 vote
41 views

how to approximate the eigendecomposition of a correlation matrix when the data have been standardized?

Context I am working to develop a penalized regression framework that will scale up to analyzing high dimensional data with a certain correlation structure. Let $X$ represent an $n \times p$ matrix of ...
31 views

Time series cross validation for trend + tree model

I have a time series data set with ~3 years of data sampled daily from 2021 to 2024. The data set exhibits a trend, and clear cycles with periods of 1 year and 1 week. My goal is to forecast ~3 ...
45 views

Help with completing a derivation of usefulness of cross-validation

This question is raised as a result of my attempt to answer this other question of mine. Let's refer to all our prior knowledge, both explicit and implicit, as $X_\text{true}$. Almost always, we are ...
• 187
23 views

Statistical Significance Testing for Nested Cross-Validation in ML Experiment

I am currently working on an ML experiment where I use a nested 5-cross validation procedure and obtain a NDCG@10 scores for each test user. I am comparing 6 different ML algorithms and have data for ...
• 101
15 views

How to split data when training and tuning the meta learner in stacking?

I have a simple yet tricky conceptual question about the data splitting of a meta learning process. Assume I have a simple X_train, ...
• 43
22 views

k-fold cross-validation for extremely imbalanced classes?

I'm doing a classification project with two imbalanced classes. I'm aware that for k-fold cross-validation in Python one can use the option "stratify" when making the splits to account for ...
33 views

• 187
30 views

Can you store the value of the predicted variable (Y) at each fold and then correlate the predicted values with the actual data?

In particular, imagine to have a set of features (X) that I use to predict a continuos variable (Y). Is it possible to use elastic net, in a cross-validation framework, use it to predict the value of ...
25 views

Is averaging RMSE values across cross-validation folds mathematically invalid?

Correct me if I'm wrong, but it seems that both scikit-learn and tidymodels will average the metrics of choice (RMSE, R2, etc.) ...
• 586
17 views

How to Train a Model on the Whole Outer-loop Training Set in Nested Cross-validation?

I'm implementing nested cross-validation for a machine learning project and need some clarity on the training process using the outer-loop training set. Here’s a summary of my process: Outer-loop ...
47 views

What can I do about model tuning parameter instability?

I am trying to determine the importance of watershed characteristics on the slope of the concentration-discharge relationship for several rivers. I am using partial least squares regression (plsr) ...
• 1
11 views

Can we use decomposed time series in a test set to obtain models accuracy metrics?

Currently I'm cross validating (rolling time window, different test lengths) several forecast models to obtain performance comparison on highly skewed time series (due to true ocassional outliers). My ...
• 491
1 vote
100 views

Hyperparameter tuning for small datasets

I have about 10 small imbalanced datasets (some of them only have about 150 samples). I want to try a bunch of balancing techniques on some models. For that, I'm using the repeated stratified cross-...
12 views

Is GroupKFold needed if some samples have some of their feature values equal?

I am given a dataset $D$ of 10k enzyme-substrate complexes having a lock-key relationship, with each sample (complex) being characterized by enzyme features $x_e$ and substrate features $x_s$. That is,...
• 477
17 views

Regression metrics calculation for each fold vs calculation at the end of k-fold cross validation [duplicate]

I stumbled upon a small realization while I was calculating fit metrics during k-fold cross-validation. Please refer to the following images: The approach of calculating R² or other fit metrics for ...
• 268
12 views

Select classification model using nested cv and bootstrap auc confidence interval

My goal is to find the best 1 model out of 55 classification models. I first ran nested cv on 55 models to see which model had better generalization. The AUC score was used as an evaluation indicator. ...
• 79
1 vote