Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [cross-validation]

Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.

0
votes
0answers
11 views

Minimum sample size for time series cross-validation (tsCV)

I am doing cross-validation of an autoregressive neural network model and I am using the tsCV function (forecast package) ...
0
votes
1answer
14 views

Scoring rules for count models on: training data vs. validation data

In order to evaluate and compare count models (e.g. Poisson regression), we can calculate scoring rules (e.g. Brier Score, Dawid-Sebastiani score, etc.) which are explained here: Error metrics for ...
0
votes
0answers
17 views

Keras GridSearchCV: Train and Validation Accuracy high but low test accuracy? (SOLVED, BUT WITH A NEW DOUBT IN CV) )

CODE SNIPPET Binary classification problem where train_data : N X H X W X F train_labels: N X 1 ...
0
votes
0answers
11 views

Time series cross validation: In some models, Training Errors are a bit higher than test errors

Following this tutorial and this question of mine, for 54 different architectures, I have created 7 fold of Time series nested cross validation and calculated their average RMSEs along the folds. ...
0
votes
0answers
11 views

Finding best kernel parameters in support vector machines

I am following this tutorial: https://www.analyticsvidhya.com/blog/2017/09/understaing-support-vector-machine-example-code/ , proposing an application of support vector machine to the well known Iris ...
0
votes
0answers
12 views

Variant of validation with singleton test sets

Is the following approach to model validation somehow reasonable? And is there a name for that approach? We have 110 data points, iid assumption holds and we want to compare two predictive models M1 ...
1
vote
0answers
28 views

Cross validation for time series prediction: How to choose the best model from different neural networks?

I want to choose the best model from a list of neural network models. My problem is a multivariate time series forecast (regression) problem, in which I forecast a parameter using other parameters, up ...
3
votes
1answer
80 views
+100

Rademacher Bound, An Alternative to Cross Validation for Ridge?

Below is a theorem from the book "Foundations of Machine Learning". It specifies the generalization bounds for Kernel Ridge Regression by making use of the Rademacher Complexity on linear models. $R(...
2
votes
1answer
14 views

Reference Request: Proof of Early Stopping Criterion

I am looking for a proof that "Validation-based early stopping" methods work but I have no idea where to start, as I am new to this field. Any recomendations of some rigerous papers that focus on ...
1
vote
1answer
26 views

Different common meanings of training set, validation set, test set [on hold]

In this previous question of mine it was pointed out in an insightful comment by ReneBT that the usage of the 3 terms training data set, validation set and test set is not uniform across the cross ...
0
votes
0answers
24 views

How to deal with Nan in machine learning (ML) algorithms using R? [closed]

I am trying to fit an ML algorithm. In particular a cubist regression model, which is a tree model where the terminal leaves contain linear regression models. I am trying to fit this model on 52 soil ...
0
votes
0answers
12 views

Best practice for presenting classifier accuracy using cross-validation?

I have a classifier whose classification accuracy I must present. Of course, I could just do e.g. a single 90/10 train/test split and report the test accuracy. However, my dataset is fairly small, so ...
0
votes
1answer
44 views

Should I Choose the best model based on test error or validation error?

I divided my dataset to training, validation and test sets. Then trained multiple forecasting models on the training dataset. now I have 3 errors for each model: Training error Validation error Test ...
-2
votes
0answers
14 views

Cross Validation Not Working [migrated]

I am trying to do cross validation for 4 different models. The goal is to predict the next 10 data. My data set contains 180 data entry. For some reason, my code is not running. ...
0
votes
2answers
29 views

Bayes factors and predictive accuracy in model comparison in rstan / brms

Despite reading up on the subject, I can't wrap my head round it, so the question remains on shaky grounds, and responses along the lines of "read chapter x" are very welcome. What I'm doing is I'm ...
0
votes
0answers
18 views

Preprocessing and hyper-parameter tuning within cross-validation

My first approach was to split the data-set into training and test set. Thereafter, I preprocessed the training set (normalizing and imputing the missing values) and used cross-validation to tune the ...
2
votes
2answers
63 views

Difference between model fitting and cross validation

I see these concepts quite often and want to see if I have the right intuitive understanding. Model fitting is when I have a set of data and fit a model (e.g. linear regression) as 'close' to the ...
0
votes
0answers
17 views

LOOCV in Caret works with Glmnet and not ElasticNet

I'm a phd student learning about different machine learning and cv methods so i apologize if this is a silly question. I have a decent understanding of lasso and am using the ...
0
votes
0answers
20 views

Controlling over-fitting in local cross-validation LightGBM

I am training a lightgbm model on a binary problem (~20% of events) with below parameters: ...
2
votes
0answers
20 views

Example and counterexample for Stone's (1977) assumption

Stone (1977) considers the problem of the choice of predicting density for $y$ given $x$ from a prescribed class of formal predicting densities $\{f(y|x,\alpha,S), \alpha \in \mathscr{A}\}$ whose ...
1
vote
0answers
22 views

Is this a typo in Stone's (1977) paper on asymptotic equivalence between AIC and LOOCV?

I am unsure about an expression in Stone's (1997) paper on asymptotic equivalence between AIC and LOOCV. Section 4., third line from the bottom of page 45 starts with $L(\theta)-1(y_i|x_i,\theta)$. ...
0
votes
0answers
8 views

Different k’s for KNN

I would like to see the knn model performance on my data for various values of k. Can I just take the same training data and compute on it the knn for different k values or should I do cross ...
0
votes
0answers
9 views

Is the problem with using early stopping with cross-validation specific to cross-validation or more generally whenever using early stopping?

I've read this answer and seem to understand, however I'm confused about why specifically cross-validation is mentioned. My understanding is that early stopping cannot be used because it leads to ...
0
votes
0answers
12 views

Ridge regression minimizing validation error

I have a question to a common deep learning problem, but I'm quite new to that field. Given a dataset of enough datas to represent a model, but we don't know whether the data is statistically ...
0
votes
1answer
20 views

How to determine N of LOOCV

In my textbook, it says that LOOCV is where $K=N$, but how do I find the value of $N$? Is it just $K-1$?
1
vote
0answers
25 views

Equivalence of AIC and LOOCV under mismatched loss functions

Under certain conditions, AIC and LOOCV (leave-one-out cross validation) are asymptotically equivalent (Stone, 1977). Stone's paper is less than 4 pages long, but quite mathy, so I turn here for some ...
0
votes
0answers
16 views

One standard error rule multiple hyperparameters

The one standard error rule for selecting the hyperparameter value after a cross-validation search for the LASSO or ridge regression's $\lambda$ is widely known and used. Is there an analog for this ...
0
votes
1answer
23 views

How are variance and bias interpreted in relation to data sets

To interpret the bias we just need the training data and the test data, since it is the measure of how far off the predicted values are from the true values(test data). But, to understand the variance ...
0
votes
0answers
12 views

Need for explicit overfit to gain optimum Leave-One-out results?

For an SVM classification with only few (~150) datapoints but ~100 features, I have created a Leave-One-Out setup in order to mimic the classificator performance on sort of unseen datapoints. From ...
2
votes
0answers
93 views

Leave-one-out cross validation in regression: R squared cannot be used - how else may model performance on unseen data be evaluated?

I have a regression problem with very few datapoints, therefore I want to use leave-one-out cross validation (effectively N-fold cross validation with N being the number of datapoints) to determine ...
1
vote
0answers
41 views

Cross-Validation on a multiple linear regression model, negative values?

I'm trying to demonstrate that, using a linear model with too many predictors, that the correlation can be artificially inflated, and that k-fold cross validation can expose overfitting. To do this, ...
1
vote
1answer
105 views

Which multivariate test on repeated k-fold cross-validation with collinearity?

I am running repeated k-fold cross validation (5x5-fold) for comparing two models based on 3 dependent numerical variables (X, Y, Z) and 4 independent categorical variables (A (two groups), B (five ...
1
vote
1answer
33 views

Is it possible to find hyperparameters and evaluate final model on test data, while training on all data?

How would I go about tuning my neural network hyperparameters, getting an estimate of its performance on unseen data, while finally training on the entire dataset? The only way I can think of is maybe ...
1
vote
1answer
29 views

what is done first balancing data or cross validation?

I want to classify imbalance data in two class and I want to use oversampling, undersampling and Synthetic data generation methods .for tuning my model i want use k-fold cross validation what should ...
0
votes
0answers
28 views

How to choose the optimum value of regularization parameter $\lambda $ to generate saprsity by using lasso-cross-validation?

I am working on an underwater sonar experiment and I have the transmitted signal sample data and received signal samples data with me. By using this data I am forming the linear model $x=\psi\theta$....
0
votes
1answer
70 views

How to choose the bandwidth of a KDE in python

Python's Sklearn module provides methods to perform Kernel Density Estimation. One of the challenges in Kernel Density Estimation is the correct choice of the kernel-bandwidth. I have come across ...
2
votes
2answers
36 views

Fitting a GARCH model and forecast using validation set approach In R

I have seperated the data into training and testing data. Then I fitted this simple garch model for training data as follows,(using rugarch package) ...
0
votes
0answers
32 views

Which of these is the proper way to select and evaluate the performance of my neural network model?

Let's say my dataset contains 100k images. I have manually created 10 neural network models each with their own set of hyperparameters. I want to select the one which will perform the best on unseen ...
0
votes
0answers
16 views

Best cross-validation procedure for binary classification

I made a realization recently that might make clear that my process for model training might not be precise. I almost always perform a train/test split early in the modeling process but I've begun ...
0
votes
1answer
37 views

Is it acceptable to test on 0.01% of the training data?

I'm doing a cross-corpus evaluation on text classification with a LinearSVM. I was wondering if it is acceptable to skew the training-testing split more than the usual 80-20% split. Specifically, I ...
0
votes
0answers
9 views

k-fold cross validation: What is the proper reward when fitting a distribution's hyperparameter? And how to combine across folds?

There is a hyper-parameter $h$ of a distribution $D_h(z)$ I am trying to fit (think like the bandwidth in KDE). The probabilistic story is as follows: $z$ comes from the distribution $D_h$, and then $...
0
votes
0answers
18 views

Cross Validation and Feature Selection with Chronological Split and Feature Preprocessing

I have a task with daily entries for which I need to do binary classification. Suppose I have 18 months of data and the model is refit every month. In addition I've got about 150 one-hot encoded ...
0
votes
0answers
9 views
12
votes
5answers
2k views

Philosophical question on logistic regression: why isn't the optimal threshold value trained?

Usually in logistic regression, we fit a model and get some predictions on the training set. We then cross-validate on those training predictions (something like here) and decide the optimal threshold ...
0
votes
0answers
18 views

Performing significance test with respect to cross validation

While performing sentiment analysis, I am trying to assess whether my approach using a novel feature set (similar to the delta-idf technique) outperforms the tf-idf metric using significance analysis. ...
1
vote
1answer
34 views

Interpretation of cross validation plot for Lasso regression

I am trying to understand the plot below generated in R (using the function cv.glmnet) which illustrates the cross validation process for picking the value of lambda in lasso regression. What are the ...
0
votes
0answers
10 views

Cross validation as adaptive classification

I want to use machine learning algorithms to learn and predict using the same dataset, as I can't get any golden standard data to construct the model with optimal parameters. I call this adaptive ...
1
vote
1answer
94 views

Which significance test for 5-fold cross validation

I have read this blog post, which states that either 5x2-fold, 10x10-fold or McNemar's test should be used for comparing two models on statistical significance, and does not suggest using ...
0
votes
0answers
4 views

The use of cross validation in Dynamic Linear Model (or state space model)

The dynamic linear model has the form as $$ y_t = m(\theta_t, x_t) + \epsilon_t ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (1)\\ \theta_{t+1} = F \theta_t + R \eta_t,~\eta_t \sim N(\mu_t,\Sigma_t) ~~~~~~(2) $$ ...
1
vote
1answer
38 views

Feature engineering with cross validation, then testing on a holdout data set?

We have 3000 samples for two classes, roughly 2000:1000. Our plan is to train a classifier on the samples but first to set aside 30% randomly selected stratified samples as a "holdout data set" for a ...