Questions tagged [cross-validation]
Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.
2,328 questions
0
votes
0answers
11 views
Minimum sample size for time series cross-validation (tsCV)
I am doing cross-validation of an autoregressive neural network model and I am using the tsCV function (forecast package) ...
0
votes
1answer
14 views
Scoring rules for count models on: training data vs. validation data
In order to evaluate and compare count models (e.g. Poisson regression), we can calculate scoring rules (e.g. Brier Score, Dawid-Sebastiani score, etc.) which are explained here: Error metrics for ...
0
votes
0answers
17 views
Keras GridSearchCV: Train and Validation Accuracy high but low test accuracy? (SOLVED, BUT WITH A NEW DOUBT IN CV) )
CODE SNIPPET
Binary classification problem where train_data : N X H X W X F
train_labels: N X 1
...
0
votes
0answers
11 views
Time series cross validation: In some models, Training Errors are a bit higher than test errors
Following this tutorial and this question of mine, for 54 different architectures, I have created 7 fold of Time series nested cross validation and calculated their average RMSEs along the folds. ...
0
votes
0answers
11 views
Finding best kernel parameters in support vector machines
I am following this tutorial: https://www.analyticsvidhya.com/blog/2017/09/understaing-support-vector-machine-example-code/ , proposing an application of support vector machine to the well known Iris ...
0
votes
0answers
12 views
Variant of validation with singleton test sets
Is the following approach to model validation somehow reasonable? And is there a name for that approach? We have 110 data points, iid assumption holds and we want to compare two predictive models M1 ...
1
vote
0answers
28 views
Cross validation for time series prediction: How to choose the best model from different neural networks?
I want to choose the best model from a list of neural network models. My problem is a multivariate time series forecast (regression) problem, in which I forecast a parameter using other parameters, up ...
3
votes
1answer
80 views
+100
Rademacher Bound, An Alternative to Cross Validation for Ridge?
Below is a theorem from the book "Foundations of Machine Learning".
It specifies the generalization bounds for Kernel Ridge Regression by making use of the Rademacher Complexity on linear models. $R(...
2
votes
1answer
14 views
Reference Request: Proof of Early Stopping Criterion
I am looking for a proof that "Validation-based early stopping" methods work but I have no idea where to start, as I am new to this field. Any recomendations of some rigerous papers that focus on ...
1
vote
1answer
26 views
Different common meanings of training set, validation set, test set [on hold]
In this previous question of mine it was pointed out in an insightful comment by ReneBT that the usage of the 3 terms training data set, validation set and test set is not uniform across the cross ...
0
votes
0answers
24 views
How to deal with Nan in machine learning (ML) algorithms using R? [closed]
I am trying to fit an ML algorithm. In particular a cubist regression model, which is a tree model where the terminal leaves contain linear regression models. I am trying to fit this model on 52 soil ...
0
votes
0answers
12 views
Best practice for presenting classifier accuracy using cross-validation?
I have a classifier whose classification accuracy I must present. Of course, I could just do e.g. a single 90/10 train/test split and report the test accuracy. However, my dataset is fairly small, so ...
0
votes
1answer
44 views
Should I Choose the best model based on test error or validation error?
I divided my dataset to training, validation and test sets. Then trained multiple forecasting models on the training dataset. now I have 3 errors for each model:
Training error
Validation error
Test ...
-2
votes
0answers
14 views
Cross Validation Not Working [migrated]
I am trying to do cross validation for 4 different models. The goal is to predict the next 10 data. My data set contains 180 data entry. For some reason, my code is not running.
...
0
votes
2answers
29 views
Bayes factors and predictive accuracy in model comparison in rstan / brms
Despite reading up on the subject, I can't wrap my head round it, so the question remains on shaky grounds, and responses along the lines of "read chapter x" are very welcome.
What I'm doing is I'm ...
0
votes
0answers
18 views
Preprocessing and hyper-parameter tuning within cross-validation
My first approach was to split the data-set into training and test set. Thereafter, I preprocessed the training set (normalizing and imputing the missing values) and used cross-validation to tune the ...
2
votes
2answers
63 views
Difference between model fitting and cross validation
I see these concepts quite often and want to see if I have the right intuitive understanding.
Model fitting is when I have a set of data and fit a model (e.g. linear regression) as 'close' to the ...
0
votes
0answers
17 views
LOOCV in Caret works with Glmnet and not ElasticNet
I'm a phd student learning about different machine learning and cv methods so i apologize if this is a silly question. I have a decent understanding of lasso and am using the ...
0
votes
0answers
20 views
Controlling over-fitting in local cross-validation LightGBM
I am training a lightgbm model on a binary problem (~20% of events) with below parameters:
...
2
votes
0answers
20 views
Example and counterexample for Stone's (1977) assumption
Stone (1977) considers the problem of the choice of predicting density for $y$ given $x$ from a prescribed class of formal predicting densities $\{f(y|x,\alpha,S), \alpha \in \mathscr{A}\}$ whose ...
1
vote
0answers
22 views
Is this a typo in Stone's (1977) paper on asymptotic equivalence between AIC and LOOCV?
I am unsure about an expression in Stone's (1997) paper on asymptotic equivalence between AIC and LOOCV. Section 4., third line from the bottom of page 45 starts with $L(\theta)-1(y_i|x_i,\theta)$. ...
0
votes
0answers
8 views
Different k’s for KNN
I would like to see the knn model performance on my data for various values of k. Can I just take the same training data and compute on it the knn for different k values or should I do cross ...
0
votes
0answers
9 views
Is the problem with using early stopping with cross-validation specific to cross-validation or more generally whenever using early stopping?
I've read this answer and seem to understand, however I'm confused about why specifically cross-validation is mentioned.
My understanding is that early stopping cannot be used because it leads to ...
0
votes
0answers
12 views
Ridge regression minimizing validation error
I have a question to a common deep learning problem, but I'm quite new to that field. Given a dataset of enough datas to represent a model, but we don't know whether the data is statistically ...
0
votes
1answer
20 views
How to determine N of LOOCV
In my textbook, it says that LOOCV is where $K=N$, but how do I find the value of $N$?
Is it just $K-1$?
1
vote
0answers
25 views
Equivalence of AIC and LOOCV under mismatched loss functions
Under certain conditions, AIC and LOOCV (leave-one-out cross validation) are asymptotically equivalent (Stone, 1977). Stone's paper is less than 4 pages long, but quite mathy, so I turn here for some ...
0
votes
0answers
16 views
One standard error rule multiple hyperparameters
The one standard error rule for selecting the hyperparameter value after a cross-validation search for the LASSO or ridge regression's $\lambda$ is widely known and used.
Is there an analog for this ...
0
votes
1answer
23 views
How are variance and bias interpreted in relation to data sets
To interpret the bias we just need the training data and the test data, since it is the measure of how far off the predicted values are from the true values(test data). But, to understand the variance ...
0
votes
0answers
12 views
Need for explicit overfit to gain optimum Leave-One-out results?
For an SVM classification with only few (~150) datapoints but ~100 features, I have created a Leave-One-Out setup in order to mimic the classificator performance on sort of unseen datapoints.
From ...
2
votes
0answers
93 views
Leave-one-out cross validation in regression: R squared cannot be used - how else may model performance on unseen data be evaluated?
I have a regression problem with very few datapoints, therefore I want to use leave-one-out cross validation (effectively N-fold cross validation with N being the number of datapoints) to determine ...
1
vote
0answers
41 views
Cross-Validation on a multiple linear regression model, negative values?
I'm trying to demonstrate that, using a linear model with too many predictors, that the correlation can be artificially inflated, and that k-fold cross validation can expose overfitting.
To do this, ...
1
vote
1answer
105 views
Which multivariate test on repeated k-fold cross-validation with collinearity?
I am running repeated k-fold cross validation (5x5-fold) for comparing two models based on 3 dependent numerical variables (X, Y, Z) and 4 independent categorical variables (A (two groups), B (five ...
1
vote
1answer
33 views
Is it possible to find hyperparameters and evaluate final model on test data, while training on all data?
How would I go about tuning my neural network hyperparameters, getting an estimate of its performance on unseen data, while finally training on the entire dataset? The only way I can think of is maybe ...
1
vote
1answer
29 views
what is done first balancing data or cross validation?
I want to classify imbalance data in two class and I want to use oversampling, undersampling and Synthetic data generation methods .for tuning my model i want use k-fold cross validation what should ...
0
votes
0answers
28 views
How to choose the optimum value of regularization parameter $\lambda $ to generate saprsity by using lasso-cross-validation?
I am working on an underwater sonar experiment and I have the transmitted signal sample data and received signal samples data with me. By using this data I am forming the linear model $x=\psi\theta$....
0
votes
1answer
70 views
How to choose the bandwidth of a KDE in python
Python's Sklearn module provides methods to perform Kernel Density Estimation.
One of the challenges in Kernel Density Estimation is the correct choice of the kernel-bandwidth.
I have come across ...
2
votes
2answers
36 views
Fitting a GARCH model and forecast using validation set approach In R
I have seperated the data into training and testing data. Then I fitted this simple garch model for training data as follows,(using rugarch package)
...
0
votes
0answers
32 views
Which of these is the proper way to select and evaluate the performance of my neural network model?
Let's say my dataset contains 100k images. I have manually created 10 neural network models each with their own set of hyperparameters. I want to select the one which will perform the best on unseen ...
0
votes
0answers
16 views
Best cross-validation procedure for binary classification
I made a realization recently that might make clear that my process for model training might not be precise.
I almost always perform a train/test split early in the modeling process but I've begun ...
0
votes
1answer
37 views
Is it acceptable to test on 0.01% of the training data?
I'm doing a cross-corpus evaluation on text classification with a LinearSVM. I was wondering if it is acceptable to skew the training-testing split more than the usual 80-20% split. Specifically, I ...
0
votes
0answers
9 views
k-fold cross validation: What is the proper reward when fitting a distribution's hyperparameter? And how to combine across folds?
There is a hyper-parameter $h$ of a distribution $D_h(z)$ I am trying to fit (think like the bandwidth in KDE). The probabilistic story is as follows:
$z$ comes from the distribution $D_h$, and then $...
0
votes
0answers
18 views
Cross Validation and Feature Selection with Chronological Split and Feature Preprocessing
I have a task with daily entries for which I need to do binary classification. Suppose I have 18 months of data and the model is refit every month. In addition I've got about 150 one-hot encoded ...
0
votes
0answers
9 views
How many folds should I use in cross-validation and is it a bad idea to restrict it?
Background
I am working with data that looks like this:
...
12
votes
5answers
2k views
Philosophical question on logistic regression: why isn't the optimal threshold value trained?
Usually in logistic regression, we fit a model and get some predictions on the training set. We then cross-validate on those training predictions (something like here) and decide the optimal threshold ...
0
votes
0answers
18 views
Performing significance test with respect to cross validation
While performing sentiment analysis, I am trying to assess whether my approach using a novel feature set (similar to the delta-idf technique) outperforms the tf-idf metric using significance analysis. ...
1
vote
1answer
34 views
Interpretation of cross validation plot for Lasso regression
I am trying to understand the plot below generated in R (using the function cv.glmnet) which illustrates the cross validation process for picking the value of lambda in lasso regression.
What are the ...
0
votes
0answers
10 views
Cross validation as adaptive classification
I want to use machine learning algorithms to learn and predict using the same dataset, as I can't get any golden standard data to construct the model with optimal parameters. I call this adaptive ...
1
vote
1answer
94 views
Which significance test for 5-fold cross validation
I have read this blog post, which states that either 5x2-fold, 10x10-fold or McNemar's test should be used for comparing two models on statistical significance, and does not suggest using ...
0
votes
0answers
4 views
The use of cross validation in Dynamic Linear Model (or state space model)
The dynamic linear model has the form as
$$
y_t = m(\theta_t, x_t) + \epsilon_t ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (1)\\
\theta_{t+1} = F \theta_t + R \eta_t,~\eta_t \sim N(\mu_t,\Sigma_t) ~~~~~~(2)
$$
...
1
vote
1answer
38 views
Feature engineering with cross validation, then testing on a holdout data set?
We have 3000 samples for two classes, roughly 2000:1000.
Our plan is to train a classifier on the samples but first to set aside 30% randomly selected stratified samples as a "holdout data set" for a ...
