Questions tagged [cross-validation]
Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.
3,469
questions
0
votes
0
answers
16
views
Model fitting with Chinese Restaurant Process
I am trying cluster a trajectory, consisting of (state, action) sequences, by assigning them to the most likely model that generated them using Chinese Restaurant Process. Basically my goal is to ...
- 51
0
votes
0
answers
13
views
Cross-validation - how to estimate mean standard error
Reading the wikipedia article on Cross-validation there is an example given for linear regression which contains the statement
If the model is correctly specified, it can be shown under mild ...
- 113
0
votes
0
answers
22
views
What exactly is the right approach when trying to find OOS MSE when using linear lasso regression?
This isn't a question where I have a code example to provide. It is more of an informal question about what to do between 2 options.
Assume I have some data and my goal is to fit a model using the ...
- 1
1
vote
1
answer
117
views
Help with Classification model for S&P500
I have started a project in order to develop my coding skills, where I am predicting next month's S&P500 return direction based on some macroeconomic and financial variables. These datasets have ...
- 13
1
vote
1
answer
27
views
Should we use train, validation, or test data when creating PR/AUC curves to optimize the decision threshold?
It makes sense to me that we can use the ROC-AUC and PR-AP scores of the validation sets during CV to tune our model hyperparameter selection. And when reporting the models final performance, it makes ...
0
votes
0
answers
27
views
Hyperparameter selection after nested cross-validation and making comparisons with DeLong's test
I have already read all the associated questions on the topic but couldn't find a clear answer. I initially split my data into training (80%) and hold-out testing (20%). Then, I am performing nested ...
0
votes
0
answers
34
views
Cross-validation and automated binning of a continuous variable for a continuous target
I am building a pipeline in a machine learning project in which I would like to automatically discretize variables containing NAs. These NAs are justified in the context of the research and it is ...
0
votes
0
answers
12
views
Why do I always get Recall=0, Precision=0 in Class 1 only in Fold1 during 5-fold validation?
I implement a classifier in python based on Negative Selection (Artificial Immune Systems) that classifies a dataset of transactions as either fraud (Class 1) or non-fraud (Class 0).
The dataset is ...
- 1
1
vote
1
answer
36
views
Benchmark machine-learning model in MLR3 with randomized data
I am conducting machine-learning in R using mlr3. I would like to assess the performance of my model by conducting a benchmark of my model using real and also randomized data.
Here is an example:
<...
- 23
0
votes
0
answers
25
views
Getting difficulties which hyperparameters I have to select (5-Fold CV with optuna to find optimal GB regression model's hyperparameters)
I'm currently searching best hyperparameters for a regression model of Gradient Boost, using 5-Fold Cross-Validation and OPTUNA.
However, I'm uncertain about which model's hyperparameters to choose, ...
- 1
0
votes
0
answers
24
views
Generalized cross-validation (GCV) with nonzero prior mean
I came across this concept of GCV optimization (new to me) for tuning hyperparameters in a model, as an alternative to maximizing the maginal likelihood (MML) of the output, which is what I am used to ...
0
votes
0
answers
23
views
Tuning hyperparameters after multiple runs
I wrote a classifier that uses LGBMClassifier. This is the code:
...
- 101
1
vote
0
answers
9
views
what does it mean to have slightly higher valid error rate, very high and similar valid error rate compared to train and test?
the dataset is split into 3 categories: train, test, valid. after training the error rates we see have these values.
case 1
1% error on the training set.
5% error on training dev set.
5% error on the ...
- 111
1
vote
1
answer
23
views
CV score vastly different from Train-Test score
I'm working on a multi-class classification task. I'm currently trying to tune a LGB model but have encountered a behavior that I do not understand. First, my data is from 1996 to 2015 so I split my ...
- 11
2
votes
2
answers
142
views
Does k-fold cross validation strictly require shuffling of data before splitting it into k groups?
Both according to Wikipedia, and to this blog post, k-fold cross validation seems to require that you shuffle the data. I have two questions:
Firstly, why? If you have a time sequence of training ...
- 674
0
votes
0
answers
14
views
Hypothesis testing for k-times-k-fold cross validation
I've done 10-times-10-fold cv over 4 datasets for 2 different machine learning classification models. So I obtain 100 performance metrics(eg. balanced accuracy) for each dataset.
I would like to ...
- 1
0
votes
1
answer
45
views
In sliding window regression, what is the best way to select my training window and test set size?
I am trying to forecast an index option's implied volatility using a sliding window regression and I'm a little confused on how I can go about cross validating with respect to the training and test ...
2
votes
1
answer
49
views
Is K-Fold cross-validation more optimistic than Bootstrap in the estimation of error?
This question comes from the basis that when running a bootstrap sample on a population whose distribution is normal, there would be higher chances for the sample to contain individuals revolving ...
0
votes
0
answers
28
views
How can I reduce fluctuations in my validation accuracy?
I'm training a CNN with pictures data for binary classification and while my training accuracy increases, my validation accuracy keeps fluctuating between small and high values of accuracy. I have a ...
- 1
5
votes
2
answers
135
views
What to do after Cross-Validation?
My question is a follow-up of posts such as this.
To summarise, I am given say 1000 data points.
I want to fit say Random Forest and optimise among $n$ choices of hyperparameters using k-fold cross-...
- 135
0
votes
0
answers
19
views
Why are my training and validation curves suspiciously close to one another (sklearn neural network)
I am trying to graph the accuracy, error and precision scores over epoch for a neural network and am using cross validation. However, my training and validation scores are practically on top of one ...
0
votes
0
answers
11
views
Where does cross-validation fit into a model selection workflow with inference?
Say we have some model, $f(x) = \hat{Y}$, such as linear regression, that estimates an output dependent value for some set of input data.
We want to do inference on the coefficients of the model to ...
0
votes
0
answers
36
views
Is it normal to have a sharp increase in validation error when using 10% of the data instead of something like 20-30%?
Scenario:
I'm training a relatively simple neural network to classify pairs of tabular datapoints (~150k), lets say drugs and diseases, whether they are related (positive) or not (negative). As I only ...
- 75
1
vote
1
answer
68
views
Is it reasonable to perform cross-validation on the test dataset?
Let’s consider the following classic model selection + performance estimation strategy for some supervised learning task.
We split our data into train and test dataset in some proportion (e.g., 25% / ...
- 41
0
votes
0
answers
20
views
Implications of Unequal Fold Sizes in Cross-Validation
I have split my dataset into k equally sized folds for cross-validation. However, I want to perform some additional sampling operations on the training set within each fold and this might make the ...
0
votes
0
answers
20
views
Nested Cross Validation performance on each Sequential Feature Selection subset
I want to get cross-validated performance values of my model after hyperparameter tuning and sequential feature selection on each feature subset. Following this example, I want to use an outer-CV ...
- 132
0
votes
1
answer
66
views
Can this be considered overfitting?
I have been trying to use the LSTM model for a monthly time series with a length of 404 (384 for training and 20 for test). I created 4 pairs of training/validation sets, trained different models, and ...
0
votes
0
answers
56
views
Training Set, Validation Set and Test Set for Time Series
I have monthly time series data on stock prices from January 1990 to August 2023. I tend to use walk-forward validation to compare the forecasting performance of ARIMA and LSTM models on this time ...
0
votes
1
answer
98
views
WOE (Weight of evidence) cross-validation bias
I have a task to create credit scoring model using WOE encoding. I have a very small dataset, so I wont be able to perform testing on test and out-of-time samples. Thus, I am going to use cross-...
- 33
1
vote
0
answers
25
views
Can cross-validation be involved in model-building rather than validation?
I have a general idea in mind that would go like this:
randomly split the data into training/testing
build a model on the training data by choosing from among candidate predictors
evaluate it on the ...
- 2,481
0
votes
0
answers
5
views
Feature Selection before Hyper-parameter Tuning and Error Estimation
I have a huge pool of features for a classification task.
For error estimation of my models, I am using Nested Cross Validation (Nested CV), where I have an inner loop of CV for hyper-parameter tuning ...
1
vote
1
answer
32
views
Classification Threshold Optimization after GridSearchCV
In my machine learning problem I am using a CNN to classify images.
Since my dataset is imbalanced I want to perform classification probability threshold tuning so I can find the optimal balance ...
- 11
0
votes
0
answers
35
views
Low CV-RMSE and negative $R^2$ (comparative)
I am trying to predict a numeric variable using XGBoost with optuna for hyperparameter optimization. I defined two objective functions for optuna, one optimized for very small datasets (5 to 17 ...
- 1
0
votes
0
answers
15
views
How to best perform early stopping and operation point choice when doing cross validation
I currently have my dataset split in three sets, training, validation, and a hold-out test set. I am training neural networks for a binary classification problem using the following protocol:
Train (...
- 115
0
votes
1
answer
17
views
Should I be concerned of Over-Optimism in Nested Cross Validation with Multiple Scoring Criteria?
Suppose I am using Nested CV to estimate the generalized error of a model and a set of hyperparameters. I have chosen two scoring metrics of interest: Brier Score and AUC, because I want to get an ...
- 23
1
vote
0
answers
62
views
Unusual results from XGBoost learning curves
I'm working on training an XGBoost classification model on time series data. Currently, I have a lot of data and it is hard to fit it all in memory, so I am trying to better understand if more data ...
- 21
0
votes
0
answers
20
views
Effective use of benchmarks for time series forecasts
I am forecasting sales of products and want to ensure that I am using benchmark models properly.
Suppose I have selected a baseline model - naive seasonal. Suppose my performance metric is WMAPE.
...
0
votes
0
answers
44
views
Cross validation misunderstanding
I have some questions about the use of CV in machine learning,
I created a post some days ago and I have been given a link to a previous post (very interesting). I also read this post (https://www.r-...
- 13
0
votes
0
answers
23
views
Kernel Density Estimation, Bandwidth Tuning, Independence, and Comparison
like many of us here, I turn to kernel density estimation when I need a nonparametric estimate of a numerical feature's distribution, and in an attempt to assume as little as possible, I usually use ...
- 121
0
votes
1
answer
38
views
Cross validation and hyperparameters tuning [duplicate]
I have few questions concerning the selection of hyperparameters for predictions in cross validation.
If I understand well, during the CV, you just create folds (inner and outer for a nested CV), and ...
- 13
1
vote
0
answers
37
views
Error variance estimation in least-squares regression
Consider a regression model:
\begin{align*}
y = f({\bf x}; {\bf b}) + e\tag{1}
\end{align*}
where $y$ is the objective variable, $\bf x$ is the explanatory variable(s), $\bf{b}$ is the model ...
- 31
0
votes
0
answers
12
views
Feature ranking during feature selection with cross-validation
I am learning about feature selection and studying the method given in Chapter 19 of the book "Applied Predictive Modelling" by Max Kuhn and Kjell Johnson.
The algorithm to perform recursive ...
- 1
0
votes
0
answers
4
views
Comparing two algorithms, one is parameter free while the other is not
I wish to compare two algorithm for subspace approximation (similar to PCA). One algorithm is parameter free, while the other is not. I use cross validation to set value to this parameter, and then ...
- 809
0
votes
1
answer
35
views
Do we only use cross-validation for hyperparameter tuning?
I am still unsure why we must use cross-validation here to validate the model, or it may be unnecessary.
Is it correct to use like indicated below? or do we have to combine it with hyperparameter ...
- 13
0
votes
1
answer
37
views
Creating Cumulative Uplift Curve in Time Series CV with Rolling Windows
I'm training a binary classifier using time series cross-validation and a rolling window approach, resulting in $k$ test sets. I want to construct a cumulative uplift curve to evaluate the model.
I've ...
- 115
0
votes
0
answers
62
views
Rolling window validation for time series classification: good idea?
I have a time series dataset (interval = 10 minutes) that contains a user's visited locations. I derive several features from the timestamps to capture the user's trend: hour of the day, day of the ...
- 103
0
votes
1
answer
23
views
Model diagnostics & Cross-Validation
I have some model which I want to cross-validate. I want to use a leave-one-out cross-validation (LOO-CV), as I have only little data. I created a model for all of my data, did model selection and ...
- 125
0
votes
0
answers
24
views
Effects of duplicate values in the subset of the predictors in ML
I am working on a classification problem with an expanded dataset. Assume that the initial data is the following. For the sake of demonstration, let us assume that we have only two predictor columns, ...
0
votes
0
answers
84
views
Training on the full dataset after cross-validation, proof, part 2
I was reading the question Training on the full dataset after cross-validation?
and found a corresponding literature reference Moiser 1951, https://doi.org/10.1177/001316445101100101 who claims good ...
- 822
0
votes
0
answers
21
views
Should I fit(X_train, y_train) or fit(X,y) when using pipeline with gridsearchCV and cross_validate? (python)
I'm trying to make a pipeline that will do the following things: preprocessing, train one model (e.g. random forest), use GridSearchCV to tune hyperparameters (using nested cross validation to prevent ...
- 1