Questions tagged [cross-validation]
Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.
1
vote
0answers
7 views
Average time series forecast errors from cross-validation with rolling origin
I'm calculating the MAPE and RMSE over a rolling origin cross-validation with fixed forecast interval for several models. For example, for a daily series with 3 years, I'm training my model with 2 ...
0
votes
0answers
14 views
Bias correction when using loo cross-validation to replace unreliable PSIS-LOO estimates
The PSIS-LOO information criterion (see this paper by Vehtari, Gelman, and Gabry) assigns a Pareto shape parameter $\hat k$ to each observation in the data, and these $\hat k$ values can be used to ...
1
vote
1answer
19 views
is permutation testing functionally equivalent to training/test?
If permutation testing is applied in machine learning, is permutation testing functionally equivalent to training/test?
1
vote
0answers
23 views
How to approach memory issues with up/down-sampling problem across millions of rows in database that can't be loaded locally? (class imbalance)
I'm faced with fairly typical class imbalance problem across a dataset with nearly 9MM rows (hard drive failures) that's not stored locally (it's in Postgres table; downloading a .csv of it is not ...
0
votes
0answers
8 views
MATLAB GPU-enabled functions list [on hold]
Is there a comprehensive list of MATLAB GPU-enabled functions? MATLAB documentation notes "hundreds of [GPU-enabled] functions in MATLAB and other toolboxes run automatically on GPU" (link1, link2, ...
0
votes
1answer
13 views
How to report results from Leave One Out Method
I have a question regarding Leave One Out Cross Validation (LOOCV):
When I use the method, I will have several regression outputs, one for each individual in my sample. How is this usually reported? ...
0
votes
0answers
10 views
Scaling time series data in a rolling cross-validation scheme
I have some time series data with a bunch of different features that I'd like to use to find the hyperparameters to a machine learning algorithm(lightgbm). Some of the features represent the prices of ...
0
votes
0answers
9 views
Stratified cross validation with groups
I have a model for a binary classification problem that I want to cross validate. The data is divided into groups. Some groups contain samples in both classes, others only contain samples from one ...
0
votes
1answer
12 views
How does train-validation-test procedure deals with the sampling error of the accuracy measure?
Let's consider a standard model selection procedure:
We have N different untrained models (for example linear regression, neural network, decision tree and so on).
We use a data set A to train each ...
0
votes
0answers
5 views
Nested k-fold cross validation: How to choose hyperparameter for a SVM
I am currently trying to understand how exactly to use nested k-fold cross validation for hyperparameter tuning / model selection. There is one aspect I really cannot get my head around.
I found ...
0
votes
0answers
24 views
Derive the estimator for the integrated squared bias $\int \left(\operatorname{E}\hat{f} - f\right)^2 $
This problem is found in p. 77 of Wand & Jones' (1995) book. If you are familiar with nonparametric estimation you may skip this introduction.
Suppose we want to minimize the integrated squared ...
0
votes
0answers
8 views
Final model from nested Cross validation
I use linear SVM and have a small dataset. Because of this I decided to so nestedCV for model checking and dir obtaining the penalty Parameter C. However, I am still confused on how to get to my final ...
1
vote
1answer
31 views
How can I create a meaningful weighting for RMSE?
Background
I should start off by saying I am not a mathematician and please excuse simple/stupid mistakes!
The goal of my exercise is to find the “best-fitting” model for the purpose of prediction. ...
1
vote
1answer
32 views
Why is bootstrapping called an “optimistic” model validator? When should I use bootstrapping or cross validation?
I understand that k-fold cross validation is a pessimistic model validator because it overestimates generalization error as less data is involved in training sets. Is bootstrapping called "optimistic" ...
0
votes
0answers
7 views
Stacking Final Model Development After Cross Validation
CV (or Nested CV) are normally done to evaluate and compare different ML algorithms as part of model development and evaluation phases. Once these stages are complete, one normally develops the final ...
1
vote
1answer
22 views
Appropriate way to get Cross Validated AUC
I was thinking about cross-validation and how it is the most appropriate way to do it...
Let's take the case of binary logistic regression where the goal is to calculate the AUC.
Make the partition ...
0
votes
0answers
8 views
How to deal with possible heterogeneity in predictive performance?
I have a nonlinear regression model that I want to fit to data with Least Absolute Deviation. The model is going to be used for predictive purposes.
The data can be divided into two subsets, that ...
1
vote
0answers
20 views
Asymptotic equivalence between cross validation and bayesian information criteria
I heard that Bayesian information criteria and cross validation are asymptotically equivalent when the size of validation set is large enough similar to the relationship between Akaike information ...
1
vote
1answer
23 views
why p value = 0 in CV-ANOVA?
i am using SIMCA 13.0 to build a opls-da model. To validate the model,SIMCA perform ANalysis Of VAriance testing of Cross-Validated predictive residuals(CV-ANOVA).Then output a table(attached). In ...
0
votes
1answer
44 views
cross-validation analysis not diagnostic
I'm using k-fold cross-validation analysis for model selection, however, it does not appear to favor any particular model. There are several variants of the models and two of them are nested within (...
1
vote
1answer
56 views
calculating overall error in k-fold cross validation
when using k-fold cross validation i thought the overall error was equal to the mean of errors of each fold. the error being anything from MAE and RMSE to NDCG,F-measure, precision and recall. however ...
0
votes
0answers
11 views
k-fold optimization in C5.0 algorithm
How can I do k-fold cross validation for C5.0 algorithm, I know caret package has createFolds function but I think in k-fold process we must do kind of averaging from all models.
Because I didn't ...
-3
votes
2answers
21 views
predictors in expected test error
In this handout by Ryan Tibshirani, it is stated that the expected test error is given by :
\begin{equation}
\mathbb{E}\Big[\frac{1}{n} \sum_{i=1}^N (y'_i-\hat{y}_i)^2\Big]
\end{equation}
where $y'_i=...
2
votes
1answer
28 views
How do I perform leave one out cross validation with boosting?
I'm working with the Anderson Iris data set and it is too small To split into a test and training set.I use boosting To make a classifier For determining the species Of flower Based on Variables in ...
0
votes
1answer
26 views
What is the reasons for a model to have a high cross validation score and yet underperforms on unseen data?
I have a model that is based on an experiment collected on 100 subjects. We are testing the model as follows:
Record raw data from the subjects
For each subject, compute the feature from the raw data ...
0
votes
0answers
9 views
Role of Nested CV as part of Model Development [duplicate]
According to my research, Nested CV is used to estimate algorithm performance without being affected by randomness introduced by hyperparameter tuning. In other words, one should do nested CV to get a ...
0
votes
0answers
41 views
How can I cross-validate a simple binary classifier?
I have a dataset of 30 observations of two variables (one is a class and it's binary, the other is a percentage and it's continuous). My ultimate goal is to build a classifier that is able to predict ...
1
vote
1answer
36 views
Classification accuracy heavily depends on random seed
I want to compare different classification methods and evaluate their prediction measures (such as accuracy etc). I first split the data into training and test set. With the training data I then ...
1
vote
0answers
30 views
When would you prefer evidence over cross-validation?
I’ve been trying to spend time learning and revising the fundamentals of model selection and performance evaluation for neural networks and linear regression.
Cross-validation seems to be the way to ...
0
votes
0answers
15 views
VAR/VEC models: checking stationarity during cross-validation
I am attempting to derive a single multivariate/vector autoregressive (VAR) model from a large dataset (6-minutes sampled at 250Hz in total w/ 50 vars) using cross-validation (CV) to optimize model-...
7
votes
1answer
100 views
Why is information about the validation data leaked if I evaluate model performance on validation data when tuning hyperparameters?
In François Chollet's Deep Learning with Python it says:
As a result, tuning the configuration of the model based on its
performance on the validation set can quickly result in overfitting to
...
0
votes
2answers
46 views
k-fold cross validation: Force at least m instances in each fold
I'm dealing with a multi-output regression problem (~ 800 dependent variables, ~ 1300 observations). My current approach is to train a single model for each output. To select an "optimal" lambda I ...
0
votes
0answers
22 views
How to model multiple training sets?
I'm analyzing the performance of 56,000 binary one-vs-all classifiers to learn how different feature sets affect object recognition performance with extremely limited training data. The data vary on ...
0
votes
0answers
24 views
RFECV + GRIDSEARCHCV on entire dataset (?)
I have 30,000 samples with 150 features for a binary classification problem, now I plan to follow: https://stackoverflow.com/questions/23815938/recursive-feature-elimination-and-grid-search-using-...
0
votes
2answers
42 views
When MSE for CV is greater than test MSE?
In an introduction to statistical learning book, on page 311 there is a figure which compares MSE vs the number of leaves.
I like to know the reason that Cross-Validation's Mean Square Error curve is ...
1
vote
0answers
34 views
What do they mean by Robust Cross-Validation?
I was reading a Kaggler Interview article and they kept specifying the importance of a stable and good cross-validation in order to win their competitions. What do they mean by that? I usually just ...
1
vote
0answers
18 views
How to validate(with sample-split data) and calibrate Cox model with time-dependent covaraites?
I am building 2 cox models:
Without time-dependent covariates
With time-dependent covariates.
1.The first model (without time-dependent variables) as specified below in R works fine and I have no ...
1
vote
1answer
57 views
How do I calculate AUC with leave-one-out CV
In a binary response setting (data matrix D with N rows) I have performed LOOCV and obtained a final lambda*. The average CV error for this lambda* is also, as I understand it, an unbiased estimator ...
0
votes
0answers
17 views
Cross-validation strategy for time-series converted to feature-space
In order to apply many machine learning classification algorithms on [multivariate] time-series, it seems necessary (and accepted) to transform the time-series to the feature-matrix space.
(e.g. https:...
0
votes
0answers
28 views
glmnet: Nested cross validation, tuning alpha and lambda
I am trying to perform the nested cross validation with glmnet and I want to tune both alpha and lambda. I want to pass the algorithm a sequence of possible alphas and let it decide for the lambda ...
6
votes
2answers
105 views
Does cross validation say anything about parsimony?
Suppose I had a set of models that all attempt to explain some phenomena. According to a sensible—and appropriately cross-validated—performance metric, all of the models perform comparably well. The ...
0
votes
1answer
29 views
Ensemble learning timeseries: Standard K-fold cross-validation ok for final step?
I have used 5 different classification models to predict future price direction (up or down) using caret's timeslice for each model type.
I now want to put all the models predicted probabilities ...
0
votes
1answer
40 views
Is it overfitting if I am using predictions from cross-validation as a level 2 feature for stacking model?
I am learning how to stack models, but I am worried if this is not a practical way to do it. I am using the full dataset and using cross_val_predict to get the ...
1
vote
1answer
37 views
How can I use linear/logistic regression for inference with colinear variables and a smallish dataset?
I have a dataset of around 120 observations, with 30 calculated variables and I am trying to predict a continuous response (result of an experiment) using those 30 variables. Ideally the smallest ...
0
votes
0answers
37 views
How does a covariance matrix over fit if we have too few data points?
I am reading this : -
Honey I shrunk the Sample Covariance Matrix
The author on page 2 says that :
The crux of the method is that those estimated coefficients in the sample covariance matrix that ...
1
vote
1answer
18 views
How can I use LOOCV to compare several different methods and measure how well they will generalize outside the N=150 sample?
I have a dataset with around N=150 people described in the dataset and I have a large amount of information about each person. I want my model to be interpretable; that is, I want to be able to ...
0
votes
0answers
20 views
How to best evaluate a cross validation of a logistic regression using cbind
I ran a logistic GLMM using cbind for the response:
...
0
votes
1answer
51 views
How to determine if one predictive model is statistically significantly better than another one?
I have a data set, two competing predictive models (regressions) and I need to decide which predictive model is better. Let us also assume that I have a measure of accuracy (for example mean squared ...
1
vote
0answers
18 views
Adjustment for binary classification with differing proportions
My data had a different proportion of 1 (20%) and 0 (80%). I found here that we can use upsampling to get a good sensitivity. The caret package in ...
1
vote
2answers
147 views
Cross-validation and building a final model when using hyperparameter optimization
I am trying to build a Gaussian process (GP) regression for a problem in which each experiment is computationally expensive, using cross-validation. Here is how I do it:
Build the GP regressor on the ...
