Questions tagged [cross-validation]

Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.

Filter by
Sorted by
Tagged with
0 votes
0 answers
16 views

Model fitting with Chinese Restaurant Process

I am trying cluster a trajectory, consisting of (state, action) sequences, by assigning them to the most likely model that generated them using Chinese Restaurant Process. Basically my goal is to ...
jumov's user avatar
  • 51
0 votes
0 answers
13 views

Cross-validation - how to estimate mean standard error

Reading the wikipedia article on Cross-validation there is an example given for linear regression which contains the statement If the model is correctly specified, it can be shown under mild ...
Peter Hull's user avatar
0 votes
0 answers
22 views

What exactly is the right approach when trying to find OOS MSE when using linear lasso regression?

This isn't a question where I have a code example to provide. It is more of an informal question about what to do between 2 options. Assume I have some data and my goal is to fit a model using the ...
Donk's user avatar
  • 1
1 vote
1 answer
117 views

Help with Classification model for S&P500

I have started a project in order to develop my coding skills, where I am predicting next month's S&P500 return direction based on some macroeconomic and financial variables. These datasets have ...
user199's user avatar
  • 13
1 vote
1 answer
27 views

Should we use train, validation, or test data when creating PR/AUC curves to optimize the decision threshold?

It makes sense to me that we can use the ROC-AUC and PR-AP scores of the validation sets during CV to tune our model hyperparameter selection. And when reporting the models final performance, it makes ...
another_student's user avatar
0 votes
0 answers
27 views

Hyperparameter selection after nested cross-validation and making comparisons with DeLong's test

I have already read all the associated questions on the topic but couldn't find a clear answer. I initially split my data into training (80%) and hold-out testing (20%). Then, I am performing nested ...
user22409235's user avatar
0 votes
0 answers
34 views

Cross-validation and automated binning of a continuous variable for a continuous target

I am building a pipeline in a machine learning project in which I would like to automatically discretize variables containing NAs. These NAs are justified in the context of the research and it is ...
hexolitemax's user avatar
0 votes
0 answers
12 views

Why do I always get Recall=0, Precision=0 in Class 1 only in Fold1 during 5-fold validation?

I implement a classifier in python based on Negative Selection (Artificial Immune Systems) that classifies a dataset of transactions as either fraud (Class 1) or non-fraud (Class 0). The dataset is ...
Mitsos 's user avatar
1 vote
1 answer
36 views

Benchmark machine-learning model in MLR3 with randomized data

I am conducting machine-learning in R using mlr3. I would like to assess the performance of my model by conducting a benchmark of my model using real and also randomized data. Here is an example: <...
user374497's user avatar
0 votes
0 answers
25 views

Getting difficulties which hyperparameters I have to select (5-Fold CV with optuna to find optimal GB regression model's hyperparameters)

I'm currently searching best hyperparameters for a regression model of Gradient Boost, using 5-Fold Cross-Validation and OPTUNA. However, I'm uncertain about which model's hyperparameters to choose, ...
Guy's user avatar
  • 1
0 votes
0 answers
24 views

Generalized cross-validation (GCV) with nonzero prior mean

I came across this concept of GCV optimization (new to me) for tuning hyperparameters in a model, as an alternative to maximizing the maginal likelihood (MML) of the output, which is what I am used to ...
smallStackBigFlow's user avatar
0 votes
0 answers
23 views

Tuning hyperparameters after multiple runs

I wrote a classifier that uses LGBMClassifier. This is the code: ...
Zag Gol's user avatar
  • 101
1 vote
0 answers
9 views

what does it mean to have slightly higher valid error rate, very high and similar valid error rate compared to train and test?

the dataset is split into 3 categories: train, test, valid. after training the error rates we see have these values. case 1 1% error on the training set. 5% error on training dev set. 5% error on the ...
ERJAN's user avatar
  • 111
1 vote
1 answer
23 views

CV score vastly different from Train-Test score

I'm working on a multi-class classification task. I'm currently trying to tune a LGB model but have encountered a behavior that I do not understand. First, my data is from 1996 to 2015 so I split my ...
jauyjad's user avatar
  • 11
2 votes
2 answers
142 views

Does k-fold cross validation strictly require shuffling of data before splitting it into k groups?

Both according to Wikipedia, and to this blog post, k-fold cross validation seems to require that you shuffle the data. I have two questions: Firstly, why? If you have a time sequence of training ...
HelloGoodbye's user avatar
0 votes
0 answers
14 views

Hypothesis testing for k-times-k-fold cross validation

I've done 10-times-10-fold cv over 4 datasets for 2 different machine learning classification models. So I obtain 100 performance metrics(eg. balanced accuracy) for each dataset. I would like to ...
fnmelo's user avatar
  • 1
0 votes
1 answer
45 views

In sliding window regression, what is the best way to select my training window and test set size?

I am trying to forecast an index option's implied volatility using a sliding window regression and I'm a little confused on how I can go about cross validating with respect to the training and test ...
sixth-sense-81's user avatar
2 votes
1 answer
49 views

Is K-Fold cross-validation more optimistic than Bootstrap in the estimation of error?

This question comes from the basis that when running a bootstrap sample on a population whose distribution is normal, there would be higher chances for the sample to contain individuals revolving ...
Renaud Bied-charreton's user avatar
0 votes
0 answers
28 views

How can I reduce fluctuations in my validation accuracy?

I'm training a CNN with pictures data for binary classification and while my training accuracy increases, my validation accuracy keeps fluctuating between small and high values of accuracy. I have a ...
KClav's user avatar
  • 1
5 votes
2 answers
135 views

What to do after Cross-Validation?

My question is a follow-up of posts such as this. To summarise, I am given say 1000 data points. I want to fit say Random Forest and optimise among $n$ choices of hyperparameters using k-fold cross-...
Smerdjakov's user avatar
0 votes
0 answers
19 views

Why are my training and validation curves suspiciously close to one another (sklearn neural network)

I am trying to graph the accuracy, error and precision scores over epoch for a neural network and am using cross validation. However, my training and validation scores are practically on top of one ...
iron_carbon's user avatar
0 votes
0 answers
11 views

Where does cross-validation fit into a model selection workflow with inference?

Say we have some model, $f(x) = \hat{Y}$, such as linear regression, that estimates an output dependent value for some set of input data. We want to do inference on the coefficients of the model to ...
Estimate the estimators's user avatar
0 votes
0 answers
36 views

Is it normal to have a sharp increase in validation error when using 10% of the data instead of something like 20-30%?

Scenario: I'm training a relatively simple neural network to classify pairs of tabular datapoints (~150k), lets say drugs and diseases, whether they are related (positive) or not (negative). As I only ...
oliver.c's user avatar
1 vote
1 answer
68 views

Is it reasonable to perform cross-validation on the test dataset?

Let’s consider the following classic model selection + performance estimation strategy for some supervised learning task. We split our data into train and test dataset in some proportion (e.g., 25% / ...
kissmemiau's user avatar
0 votes
0 answers
20 views

Implications of Unequal Fold Sizes in Cross-Validation

I have split my dataset into k equally sized folds for cross-validation. However, I want to perform some additional sampling operations on the training set within each fold and this might make the ...
Fernando Rubio Garcia's user avatar
0 votes
0 answers
20 views

Nested Cross Validation performance on each Sequential Feature Selection subset

I want to get cross-validated performance values of my model after hyperparameter tuning and sequential feature selection on each feature subset. Following this example, I want to use an outer-CV ...
Charlie's user avatar
  • 132
0 votes
1 answer
66 views

Can this be considered overfitting?

I have been trying to use the LSTM model for a monthly time series with a length of 404 (384 for training and 20 for test). I created 4 pairs of training/validation sets, trained different models, and ...
Mamad Fasih's user avatar
0 votes
0 answers
56 views

Training Set, Validation Set and Test Set for Time Series

I have monthly time series data on stock prices from January 1990 to August 2023. I tend to use walk-forward validation to compare the forecasting performance of ARIMA and LSTM models on this time ...
Mamad Fasih's user avatar
0 votes
1 answer
98 views

WOE (Weight of evidence) cross-validation bias

I have a task to create credit scoring model using WOE encoding. I have a very small dataset, so I wont be able to perform testing on test and out-of-time samples. Thus, I am going to use cross-...
FedorT54's user avatar
1 vote
0 answers
25 views

Can cross-validation be involved in model-building rather than validation?

I have a general idea in mind that would go like this: randomly split the data into training/testing build a model on the training data by choosing from among candidate predictors evaluate it on the ...
Dave's user avatar
  • 2,481
0 votes
0 answers
5 views

Feature Selection before Hyper-parameter Tuning and Error Estimation

I have a huge pool of features for a classification task. For error estimation of my models, I am using Nested Cross Validation (Nested CV), where I have an inner loop of CV for hyper-parameter tuning ...
Mehdi Ejtehadi's user avatar
1 vote
1 answer
32 views

Classification Threshold Optimization after GridSearchCV

In my machine learning problem I am using a CNN to classify images. Since my dataset is imbalanced I want to perform classification probability threshold tuning so I can find the optimal balance ...
Throwaway123's user avatar
0 votes
0 answers
35 views

Low CV-RMSE and negative $R^2$ (comparative)

I am trying to predict a numeric variable using XGBoost with optuna for hyperparameter optimization. I defined two objective functions for optuna, one optimized for very small datasets (5 to 17 ...
RaduIoan's user avatar
0 votes
0 answers
15 views

How to best perform early stopping and operation point choice when doing cross validation

I currently have my dataset split in three sets, training, validation, and a hold-out test set. I am training neural networks for a binary classification problem using the following protocol: Train (...
Alb's user avatar
  • 115
0 votes
1 answer
17 views

Should I be concerned of Over-Optimism in Nested Cross Validation with Multiple Scoring Criteria?

Suppose I am using Nested CV to estimate the generalized error of a model and a set of hyperparameters. I have chosen two scoring metrics of interest: Brier Score and AUC, because I want to get an ...
user7351362's user avatar
1 vote
0 answers
62 views

Unusual results from XGBoost learning curves

I'm working on training an XGBoost classification model on time series data. Currently, I have a lot of data and it is hard to fit it all in memory, so I am trying to better understand if more data ...
Ted's user avatar
  • 21
0 votes
0 answers
20 views

Effective use of benchmarks for time series forecasts

I am forecasting sales of products and want to ensure that I am using benchmark models properly. Suppose I have selected a baseline model - naive seasonal. Suppose my performance metric is WMAPE. ...
Azamat Bagatov's user avatar
0 votes
0 answers
44 views

Cross validation misunderstanding

I have some questions about the use of CV in machine learning, I created a post some days ago and I have been given a link to a previous post (very interesting). I also read this post (https://www.r-...
Nicolas's user avatar
  • 13
0 votes
0 answers
23 views

Kernel Density Estimation, Bandwidth Tuning, Independence, and Comparison

like many of us here, I turn to kernel density estimation when I need a nonparametric estimate of a numerical feature's distribution, and in an attempt to assume as little as possible, I usually use ...
user3163829's user avatar
0 votes
1 answer
38 views

Cross validation and hyperparameters tuning [duplicate]

I have few questions concerning the selection of hyperparameters for predictions in cross validation. If I understand well, during the CV, you just create folds (inner and outer for a nested CV), and ...
Nicolas's user avatar
  • 13
1 vote
0 answers
37 views

Error variance estimation in least-squares regression

Consider a regression model: \begin{align*} y = f({\bf x}; {\bf b}) + e\tag{1} \end{align*} where $y$ is the objective variable, $\bf x$ is the explanatory variable(s), $\bf{b}$ is the model ...
GreatJourney's user avatar
0 votes
0 answers
12 views

Feature ranking during feature selection with cross-validation

I am learning about feature selection and studying the method given in Chapter 19 of the book "Applied Predictive Modelling" by Max Kuhn and Kjell Johnson. The algorithm to perform recursive ...
seb's user avatar
  • 1
0 votes
0 answers
4 views

Comparing two algorithms, one is parameter free while the other is not

I wish to compare two algorithm for subspace approximation (similar to PCA). One algorithm is parameter free, while the other is not. I use cross validation to set value to this parameter, and then ...
Roy's user avatar
  • 809
0 votes
1 answer
35 views

Do we only use cross-validation for hyperparameter tuning?

I am still unsure why we must use cross-validation here to validate the model, or it may be unnecessary. Is it correct to use like indicated below? or do we have to combine it with hyperparameter ...
vdu16's user avatar
  • 13
0 votes
1 answer
37 views

Creating Cumulative Uplift Curve in Time Series CV with Rolling Windows

I'm training a binary classifier using time series cross-validation and a rolling window approach, resulting in $k$ test sets. I want to construct a cumulative uplift curve to evaluate the model. I've ...
Amit S's user avatar
  • 115
0 votes
0 answers
62 views

Rolling window validation for time series classification: good idea?

I have a time series dataset (interval = 10 minutes) that contains a user's visited locations. I derive several features from the timestamps to capture the user's trend: hour of the day, day of the ...
sander's user avatar
  • 103
0 votes
1 answer
23 views

Model diagnostics & Cross-Validation

I have some model which I want to cross-validate. I want to use a leave-one-out cross-validation (LOO-CV), as I have only little data. I created a model for all of my data, did model selection and ...
Zoe's user avatar
  • 125
0 votes
0 answers
24 views

Effects of duplicate values in the subset of the predictors in ML

I am working on a classification problem with an expanded dataset. Assume that the initial data is the following. For the sake of demonstration, let us assume that we have only two predictor columns, ...
ElonMuskofBadIdeas's user avatar
0 votes
0 answers
84 views

Training on the full dataset after cross-validation, proof, part 2

I was reading the question Training on the full dataset after cross-validation? and found a corresponding literature reference Moiser 1951, https://doi.org/10.1177/001316445101100101 who claims good ...
Ggjj11's user avatar
  • 822
0 votes
0 answers
21 views

Should I fit(X_train, y_train) or fit(X,y) when using pipeline with gridsearchCV and cross_validate? (python)

I'm trying to make a pipeline that will do the following things: preprocessing, train one model (e.g. random forest), use GridSearchCV to tune hyperparameters (using nested cross validation to prevent ...
wick's user avatar
  • 1

1
2 3 4 5
70