Questions tagged [cross-validation]

Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.

Filter by
Sorted by
Tagged with
0 votes
0 answers
5 views

sklearn.model_selection.cross_validate function random split? [closed]

For the sklearn.model_selection.cross_validate, when I write a loop to pass on difference model to the cross validate, like ...
user avatar
0 votes
0 answers
17 views

Conceptual question: Evaluate model performance

What I think I understood about evaluating model performance: When a model is fit on some data, the performance of this model on the same data is not a good indicator of how the model would fit on ...
user avatar
  • 525
1 vote
0 answers
10 views

Reliably evaluate model performance with very few positive samples

I do a binary classification in the domain of predictive maintenance. Setup My dataset is highly imbalanced with only 17 samples of the positive class, but an nearly indefinite amount of negative ...
user avatar
  • 525
2 votes
1 answer
37 views

Does Cross Validation Reduces Underfitting

I know that Cross-Validation helps reduce Overfitting in the data by its property of testing and training on all the instances. But is there any chance that it also reduces Underfitting? I mean What I ...
user avatar
  • 21
0 votes
0 answers
11 views

Is there such a thing as a "Random Forest of Random Forests"? [duplicate]

This was a question I had and I have tried to find an answer for a while but without any conclusive answers. In short - can you create a random forest in which each "tree" in the "...
user avatar
  • 5,874
0 votes
0 answers
26 views

Cross Validation - leave one out or k-fold?

I have read the disadvantage/advantages of using leave-one-out cross validation, but I have not been able to find anything telling how to choose if to use it. I have got a very small dataset, made out ...
user avatar
  • 109
0 votes
0 answers
16 views

(XGBOOST) 5-fold cross validation test aucpr is always lower than train aucpr, is that overfitting?

I am using XGBOOST to construct a prediction model, but no matter what I do (including set gamma, subsample, eta), I will get results similar to the following picture. I did see all train and test ...
user avatar
1 vote
0 answers
19 views

Should I use cross validation for simple linear regression model?

I have a data set with 181 observations. I have 9 predictors and I have developed different regression models using ordinary linear regression and stepwise linear regression. Now I'm trying to decide ...
user avatar
  • 11
0 votes
0 answers
10 views

Displaying Cross validated R square in dredge output [closed]

I am comparing different linear models from dredge with the following commands : globalmodel<-lm(response ~ . , data=dataset) ...
user avatar
  • 1
1 vote
0 answers
6 views

CT-Scan Classification Overfitting Problem

I am currently trying to train pretrained convolutional neural networks trained on the imagenet dataset to be able to classify ct-scans into two classes. Viral Pneumonia and Normal. I am using K-fold ...
user avatar
1 vote
1 answer
29 views

What are "volatile" learning curves indicative of? [duplicate]

I have a dataset set with ~40 features onto which I'm applying a multi-layer perceptron for regression purposes. The train, validation, and test sets are made up of 3M, 800K, and 800K examples each, ...
user avatar
  • 565
0 votes
0 answers
8 views

PRESS statistic and k-fold cross-validation

How is the PRESS statistic calculated in a k-fold cross-validation? I know how it is done in the leave-one-out scenario. Is it still summed over all training samples, just that there are now k-many ...
user avatar
0 votes
1 answer
18 views

Cross Validation on whole data for model comparision

I have good an imbalanced very small dataset (58 instances) and whould like to create a multiclass classification model. I'd like to use cross-validation in order to make the most out of the data I ...
user avatar
  • 109
1 vote
1 answer
21 views

Splitting medical dataset by patient

I am currently trying to train a CNN model to classify CT-scans. I split the dataset using K-fold cross-validation and since the dataset I am using contains multiple slices per patient, I split the ...
user avatar
1 vote
0 answers
19 views

Why isn't RandomSearchCV returning the optimum parameters for the XGBoost Model, and how can I avoid Overfitting?

I have a dataset for energy consumer customers and binary target variables with which I want to predict the churn for the customers. Counts of target values Not Churn 0: 14153 Churn 1: 1520 I have ...
user avatar
  • 31
1 vote
0 answers
45 views

Out-of-sample prediction error using nested cross-validation

I am applying Lasso regression and the R function glmnet::cv.glmnet() to obtain a prediction model based on 90% of the data. I have set aside 10% as a hold-out set and obtain predicted probabilities ...
user avatar
  • 11
5 votes
3 answers
460 views

Model Selection: AIC/BIC and Cross-Validation gives different conclusion

In general, there vast number of ways to select model/feature in machine learning or statistics. For example, empirical method like Cross-Validation, Bootstrap methods or in sample penalty such as AIC,...
user avatar
  • 404
0 votes
0 answers
10 views

Approximately Unbiased P-value vs Bootstrap Probability: which one should i choose?

Some references first: How is approximately unbiased bootstrap better than a regular bootstrap with regards to hierarchical clustering? Suzuki et al. 2004 https://www.researchgate.net/publication/...
user avatar
  • 1
1 vote
1 answer
31 views

Is cross-validation with no data leakage sufficient to replace train-test split?

I would like to seek expert advice on the topic above. I was taught to follow this workflow: Split dataset into training and testing Use training dataset to develop model Set hyperparameter in model ...
user avatar
0 votes
1 answer
26 views

Perform Rose Method, then Logistic Regression and do k -fold cross validation

I have unbalanced data so I want to oversample obs from the minority class and then apply Logistic regression to the training set. After that, I would like to perform cross-validation. My question is: ...
user avatar
  • 3
0 votes
1 answer
14 views

use Cross-validation to find the better model

...
user avatar
0 votes
0 answers
16 views

Implementing a bandwidth selection algorithm with cross validation

I'm trying to implement a bandwidth selection algorithm based on cross-validation for kernel density estimation. I learned from the article that cross-validation needs to minimize Mean Integrated ...
user avatar
  • 527
1 vote
0 answers
9 views

What error estimates is used for 5x2 cv paired t-test?

I am performing a 5x2 cv paired t-test. I have read most of the paper "Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms" by Remco R. Bouckaert and Eibe ...
user avatar
3 votes
1 answer
43 views

How to handle outcome variables during imputation of missing data in model building and assessment process?

Der community I have a question about the appropriate handling of the imputation of missing data to get an unbiased estimate of prediction accuracy during model building and assessment. While ...
user avatar
  • 31
2 votes
1 answer
50 views

Is "sensitivity at fixed specificity" a valid metric for comparing different classifiers?

For a given dataset, a common way to compare 2 classifiers is to compare their average validation accuracies using cross-validation. Is it valid to replace the accuracy with other classification ...
user avatar
  • 935
0 votes
0 answers
16 views

How to compare 2 classifiers using a classification metric?

Let's assume we have 2 binary classifiers (A and B) and some labeled dataset, and we want to compare A and B. Let's assume we use the ROC AUC as the metric (although it could be the accuracy or ...
user avatar
  • 935
0 votes
2 answers
36 views

KFold Cross Validation with Train/Test/Validation-Set

I want to do KFold Cross Validation on a specific model and I am wondering what data to use. In my project I have got a Train, Test and Validation set (this was already provided). Now I want to to ...
user avatar
1 vote
1 answer
23 views

Validation in deep learning networks

https://es.mathworks.com/help/deeplearning/ref/trainnetwork.html?s_tid=doc_ta#bu6sn4c-traininfo In the above link, there is an example for train network with augmented images. The number of iterations ...
user avatar
  • 47
0 votes
0 answers
33 views

Quantifying the difference between micro and macro averaging in cross-validation

I am studying the effect of micro vs macro averaging the results of a cross-validation run. Using true positive rate as a running example, given a cross-validation run of $K$ folds, the final $TPR$ ...
user avatar
1 vote
0 answers
20 views

Cross-validation on the HP filter for penalty selection

I'm looking for a bit more clarification on the selection of the $\lambda$ value for the Hodrick-Prescott Filter, aka HP filter. Which stems from a time-series in the form of $$y_t=g_t+\epsilon_t$$ ...
user avatar
2 votes
1 answer
33 views

Using cross validation to actually fit the final model [closed]

I usually use cross validation only for the tuning part and once I have my hyperparameters, I fit the final model using the actual model with these hyperparameters like this: ...
user avatar
  • 45
1 vote
2 answers
61 views

Cross Validation for Time Series Classification (Not Forecasting!)

Is it possible to use regular k-fold cross validation where the folds contain entire time series in time series classification? I'm asking because most sources discussing cross validation with time ...
user avatar
0 votes
0 answers
17 views

Can I perform hyperparameter tuning inside K-fold CV?

I am performing leave one subject out cross-validation, using one subject as the independent final test set to get the performance of my model. Can I perform hyperparameter tuning inside each of my K-...
user avatar
0 votes
1 answer
71 views

What is the uncertainty of Leave-one-out-cross-validation method?

I have used the LOOCV to validate my model. As we know, the LOOCV method is a special case of cross-validation where the number of folds equals the number of instances in the data set. Thus, the ...
user avatar
0 votes
0 answers
22 views

Decision tree feature importances on a test set

Many tree based models come with a built in feature importance method usually based on impurity decrease (ex. sklearn RandomForest). Would it be possible to calculate feature importances on a test set ...
user avatar
  • 1
1 vote
1 answer
34 views

Difference between cross validation vs model accuracy measures

I have a time series ARIMA model and I want to validate the accuracy my prediction. But I dont understand the difference of using cross validation vs model accuracy measures such as MAPE, MAE, MSE and ...
user avatar
  • 11
2 votes
0 answers
41 views

What is the bias adjustment needed on k-folds CV? (made by cv.glm on R's boot package)

I'm trying to find the equation defined on the documentation of the cv.glm function in the boot package: "When $K$ is less ...
user avatar
  • 21
2 votes
1 answer
32 views

Does using grid search for hyperparemeters make test set redundant?

The purpose of train, validate and test data splits addresses the issue of data leakage when tuning for the model's hyperparameters. Does Grid Search then eliminates the need for test set? Because ...
user avatar
  • 21
3 votes
0 answers
21 views

How to manage out of sample data in the long run?

For example, you are interested in testing an investment strategy and there is data from 1950 to 2022. So you split it into a train and test set, say 1950-2000 and 2000-2022. Then you build your model ...
user avatar
0 votes
1 answer
22 views

References for cross-validation implementations in Pytorch

I'm interested in good references on cross-validation implementations for feed-forward neural networks in pytorch from scratch. Thanks in advance.
user avatar
0 votes
0 answers
10 views

Are stacked meta models for time series forecasting still considered forecasters?

I am working on building a meta model that is based on the principle of stacked generalisation (1). In a nutshell, this method works by using building a meta model based on the predictions of various ...
user avatar
1 vote
1 answer
62 views

Significance test for comparing different 10-fold cross-validated Machine Learning Regressions

Is there a recommended significance test for comparing different 10-fold cross validated regressions? For instance, I want to compare the performance of LASSO against Random Forest for my dataset. ...
user avatar
0 votes
0 answers
20 views

Difference K-fold versus Blocked Cross-Validation?

In the paper "Evaluating time series forecasting models: an empirical study on performance estimation methods" by Cerqueira et al (2020), they mention k-fold cross-validation. Which they ...
user avatar
0 votes
0 answers
17 views

learning curve for RF and LR comparison and selection

I am plotting learning curve to check how the model perform on training data set and the effect of the training size on the accuracy. I am using two models, random forest and logistic regression. From ...
user avatar
4 votes
2 answers
297 views

Stratification of the continuous y (target) variable in regression setting

Is it wise to stratify the continuous y (target) variable when you split your training and testing data from the total sample in regression setting? Here is the approach in python to do implement ...
user avatar
1 vote
0 answers
18 views

Interpreting learning curves

There is really few examples online regarding interpreting learning curves and they are all of different type.It is quite confusing to me honestly.May I just ask: How should we interpret them?What ...
user avatar
0 votes
0 answers
13 views

After training a model, how does test set error inform decision making?

I split a data set into three subsets: training, validation, and test sets. I use my training data for fitting and validation to check for overfitting. I then have a final model that I then propose to ...
user avatar
  • 1,063
2 votes
1 answer
83 views

Should folds in k-fold CV actually be representative?

I have read somewhere that the k of the k-fold CV should be picked in such a manner as to have representative validation sets (folds). This seems to me to be contradictory since the leave-one-out CV ...
user avatar
0 votes
1 answer
55 views

Forcing covariates to always be part of a Lasso model

I want to use a Lasso to predict outcomes for different policy scenarios. At the optimal degree of regularization obtained by cross-validation, one important variable in whose impact I'm interested in ...
user avatar
  • 1
2 votes
0 answers
16 views

Statistical test for temporal cross validation

I estimated the performance of my forecasting model and that of a baseline on 10 folds using temporal cross-validation. With which test do I assess if my model is significantly better than the ...
user avatar
  • 423

1
2 3 4 5
65