Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [cross-validation]

Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.

1
vote
0answers
7 views

Average time series forecast errors from cross-validation with rolling origin

I'm calculating the MAPE and RMSE over a rolling origin cross-validation with fixed forecast interval for several models. For example, for a daily series with 3 years, I'm training my model with 2 ...
0
votes
0answers
14 views

Bias correction when using loo cross-validation to replace unreliable PSIS-LOO estimates

The PSIS-LOO information criterion (see this paper by Vehtari, Gelman, and Gabry) assigns a Pareto shape parameter $\hat k$ to each observation in the data, and these $\hat k$ values can be used to ...
1
vote
1answer
19 views

is permutation testing functionally equivalent to training/test?

If permutation testing is applied in machine learning, is permutation testing functionally equivalent to training/test?
1
vote
0answers
23 views

How to approach memory issues with up/down-sampling problem across millions of rows in database that can't be loaded locally? (class imbalance)

I'm faced with fairly typical class imbalance problem across a dataset with nearly 9MM rows (hard drive failures) that's not stored locally (it's in Postgres table; downloading a .csv of it is not ...
0
votes
0answers
8 views

MATLAB GPU-enabled functions list [on hold]

Is there a comprehensive list of MATLAB GPU-enabled functions? MATLAB documentation notes "hundreds of [GPU-enabled] functions in MATLAB and other toolboxes run automatically on GPU" (link1, link2, ...
0
votes
1answer
13 views

How to report results from Leave One Out Method

I have a question regarding Leave One Out Cross Validation (LOOCV): When I use the method, I will have several regression outputs, one for each individual in my sample. How is this usually reported? ...
0
votes
0answers
10 views

Scaling time series data in a rolling cross-validation scheme

I have some time series data with a bunch of different features that I'd like to use to find the hyperparameters to a machine learning algorithm(lightgbm). Some of the features represent the prices of ...
0
votes
0answers
9 views

Stratified cross validation with groups

I have a model for a binary classification problem that I want to cross validate. The data is divided into groups. Some groups contain samples in both classes, others only contain samples from one ...
0
votes
1answer
12 views

How does train-validation-test procedure deals with the sampling error of the accuracy measure?

Let's consider a standard model selection procedure: We have N different untrained models (for example linear regression, neural network, decision tree and so on). We use a data set A to train each ...
0
votes
0answers
5 views

Nested k-fold cross validation: How to choose hyperparameter for a SVM

I am currently trying to understand how exactly to use nested k-fold cross validation for hyperparameter tuning / model selection. There is one aspect I really cannot get my head around. I found ...
0
votes
0answers
24 views

Derive the estimator for the integrated squared bias $\int \left(\operatorname{E}\hat{f} - f\right)^2 $

This problem is found in p. 77 of Wand & Jones' (1995) book. If you are familiar with nonparametric estimation you may skip this introduction. Suppose we want to minimize the integrated squared ...
0
votes
0answers
8 views

Final model from nested Cross validation

I use linear SVM and have a small dataset. Because of this I decided to so nestedCV for model checking and dir obtaining the penalty Parameter C. However, I am still confused on how to get to my final ...
1
vote
1answer
31 views

How can I create a meaningful weighting for RMSE?

Background I should start off by saying I am not a mathematician and please excuse simple/stupid mistakes! The goal of my exercise is to find the “best-fitting” model for the purpose of prediction. ...
1
vote
1answer
32 views

Why is bootstrapping called an “optimistic” model validator? When should I use bootstrapping or cross validation?

I understand that k-fold cross validation is a pessimistic model validator because it overestimates generalization error as less data is involved in training sets. Is bootstrapping called "optimistic" ...
0
votes
0answers
7 views

Stacking Final Model Development After Cross Validation

CV (or Nested CV) are normally done to evaluate and compare different ML algorithms as part of model development and evaluation phases. Once these stages are complete, one normally develops the final ...
1
vote
1answer
22 views

Appropriate way to get Cross Validated AUC

I was thinking about cross-validation and how it is the most appropriate way to do it... Let's take the case of binary logistic regression where the goal is to calculate the AUC. Make the partition ...
0
votes
0answers
8 views

How to deal with possible heterogeneity in predictive performance?

I have a nonlinear regression model that I want to fit to data with Least Absolute Deviation. The model is going to be used for predictive purposes. The data can be divided into two subsets, that ...
1
vote
0answers
20 views

Asymptotic equivalence between cross validation and bayesian information criteria

I heard that Bayesian information criteria and cross validation are asymptotically equivalent when the size of validation set is large enough similar to the relationship between Akaike information ...
1
vote
1answer
23 views

why p value = 0 in CV-ANOVA?

i am using SIMCA 13.0 to build a opls-da model. To validate the model,SIMCA perform ANalysis Of VAriance testing of Cross-Validated predictive residuals(CV-ANOVA).Then output a table(attached). In ...
0
votes
1answer
44 views

cross-validation analysis not diagnostic

I'm using k-fold cross-validation analysis for model selection, however, it does not appear to favor any particular model. There are several variants of the models and two of them are nested within (...
1
vote
1answer
56 views

calculating overall error in k-fold cross validation

when using k-fold cross validation i thought the overall error was equal to the mean of errors of each fold. the error being anything from MAE and RMSE to NDCG,F-measure, precision and recall. however ...
0
votes
0answers
11 views

k-fold optimization in C5.0 algorithm

How can I do k-fold cross validation for C5.0 algorithm, I know caret package has createFolds function but I think in k-fold process we must do kind of averaging from all models. Because I didn't ...
-3
votes
2answers
21 views

predictors in expected test error

In this handout by Ryan Tibshirani, it is stated that the expected test error is given by : \begin{equation} \mathbb{E}\Big[\frac{1}{n} \sum_{i=1}^N (y'_i-\hat{y}_i)^2\Big] \end{equation} where $y'_i=...
2
votes
1answer
28 views

How do I perform leave one out cross validation with boosting?

I'm working with the Anderson Iris data set and it is too small To split into a test and training set.I use boosting To make a classifier For determining the species Of flower Based on Variables in ...
0
votes
1answer
26 views

What is the reasons for a model to have a high cross validation score and yet underperforms on unseen data?

I have a model that is based on an experiment collected on 100 subjects. We are testing the model as follows: Record raw data from the subjects For each subject, compute the feature from the raw data ...
0
votes
0answers
9 views

Role of Nested CV as part of Model Development [duplicate]

According to my research, Nested CV is used to estimate algorithm performance without being affected by randomness introduced by hyperparameter tuning. In other words, one should do nested CV to get a ...
0
votes
0answers
41 views

How can I cross-validate a simple binary classifier?

I have a dataset of 30 observations of two variables (one is a class and it's binary, the other is a percentage and it's continuous). My ultimate goal is to build a classifier that is able to predict ...
1
vote
1answer
36 views

Classification accuracy heavily depends on random seed

I want to compare different classification methods and evaluate their prediction measures (such as accuracy etc). I first split the data into training and test set. With the training data I then ...
1
vote
0answers
30 views

When would you prefer evidence over cross-validation?

I’ve been trying to spend time learning and revising the fundamentals of model selection and performance evaluation for neural networks and linear regression. Cross-validation seems to be the way to ...
0
votes
0answers
15 views

VAR/VEC models: checking stationarity during cross-validation

I am attempting to derive a single multivariate/vector autoregressive (VAR) model from a large dataset (6-minutes sampled at 250Hz in total w/ 50 vars) using cross-validation (CV) to optimize model-...
7
votes
1answer
100 views

Why is information about the validation data leaked if I evaluate model performance on validation data when tuning hyperparameters?

In François Chollet's Deep Learning with Python it says: As a result, tuning the configuration of the model based on its performance on the validation set can quickly result in overfitting to ...
0
votes
2answers
46 views

k-fold cross validation: Force at least m instances in each fold

I'm dealing with a multi-output regression problem (~ 800 dependent variables, ~ 1300 observations). My current approach is to train a single model for each output. To select an "optimal" lambda I ...
0
votes
0answers
22 views

How to model multiple training sets?

I'm analyzing the performance of 56,000 binary one-vs-all classifiers to learn how different feature sets affect object recognition performance with extremely limited training data. The data vary on ...
0
votes
0answers
24 views

RFECV + GRIDSEARCHCV on entire dataset (?)

I have 30,000 samples with 150 features for a binary classification problem, now I plan to follow: https://stackoverflow.com/questions/23815938/recursive-feature-elimination-and-grid-search-using-...
0
votes
2answers
42 views

When MSE for CV is greater than test MSE?

In an introduction to statistical learning book, on page 311 there is a figure which compares MSE vs the number of leaves. I like to know the reason that Cross-Validation's Mean Square Error curve is ...
1
vote
0answers
34 views

What do they mean by Robust Cross-Validation?

I was reading a Kaggler Interview article and they kept specifying the importance of a stable and good cross-validation in order to win their competitions. What do they mean by that? I usually just ...
1
vote
0answers
18 views

How to validate(with sample-split data) and calibrate Cox model with time-dependent covaraites?

I am building 2 cox models: Without time-dependent covariates With time-dependent covariates. 1.The first model (without time-dependent variables) as specified below in R works fine and I have no ...
1
vote
1answer
57 views

How do I calculate AUC with leave-one-out CV

In a binary response setting (data matrix D with N rows) I have performed LOOCV and obtained a final lambda*. The average CV error for this lambda* is also, as I understand it, an unbiased estimator ...
0
votes
0answers
17 views

Cross-validation strategy for time-series converted to feature-space

In order to apply many machine learning classification algorithms on [multivariate] time-series, it seems necessary (and accepted) to transform the time-series to the feature-matrix space. (e.g. https:...
0
votes
0answers
28 views

glmnet: Nested cross validation, tuning alpha and lambda

I am trying to perform the nested cross validation with glmnet and I want to tune both alpha and lambda. I want to pass the algorithm a sequence of possible alphas and let it decide for the lambda ...
6
votes
2answers
105 views

Does cross validation say anything about parsimony?

Suppose I had a set of models that all attempt to explain some phenomena. According to a sensible—and appropriately cross-validated—performance metric, all of the models perform comparably well. The ...
0
votes
1answer
29 views

Ensemble learning timeseries: Standard K-fold cross-validation ok for final step?

I have used 5 different classification models to predict future price direction (up or down) using caret's timeslice for each model type. I now want to put all the models predicted probabilities ...
0
votes
1answer
40 views

Is it overfitting if I am using predictions from cross-validation as a level 2 feature for stacking model?

I am learning how to stack models, but I am worried if this is not a practical way to do it. I am using the full dataset and using cross_val_predict to get the ...
1
vote
1answer
37 views

How can I use linear/logistic regression for inference with colinear variables and a smallish dataset?

I have a dataset of around 120 observations, with 30 calculated variables and I am trying to predict a continuous response (result of an experiment) using those 30 variables. Ideally the smallest ...
0
votes
0answers
37 views

How does a covariance matrix over fit if we have too few data points?

I am reading this : - Honey I shrunk the Sample Covariance Matrix The author on page 2 says that : The crux of the method is that those estimated coefficients in the sample covariance matrix that ...
1
vote
1answer
18 views

How can I use LOOCV to compare several different methods and measure how well they will generalize outside the N=150 sample?

I have a dataset with around N=150 people described in the dataset and I have a large amount of information about each person. I want my model to be interpretable; that is, I want to be able to ...
0
votes
0answers
20 views

How to best evaluate a cross validation of a logistic regression using cbind

I ran a logistic GLMM using cbind for the response: ...
0
votes
1answer
51 views

How to determine if one predictive model is statistically significantly better than another one?

I have a data set, two competing predictive models (regressions) and I need to decide which predictive model is better. Let us also assume that I have a measure of accuracy (for example mean squared ...
1
vote
0answers
18 views

Adjustment for binary classification with differing proportions

My data had a different proportion of 1 (20%) and 0 (80%). I found here that we can use upsampling to get a good sensitivity. The caret package in ...
1
vote
2answers
147 views

Cross-validation and building a final model when using hyperparameter optimization

I am trying to build a Gaussian process (GP) regression for a problem in which each experiment is computationally expensive, using cross-validation. Here is how I do it: Build the GP regressor on the ...