Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
10 views

cross-validation: what is the standard deviation if the same value is obtained for each fold?

Here is a detailed imaginary example: I am using 5-fold cross-validation to estimate the generalization MSE of my predictive model. When I hold-out fold number 1, which contains 10 observations, say ...
0
votes
0answers
3 views

Can a cross validation be used structurally to assess the transferability of an effect of one factor across another factor?

A MANOVA performed on multivariate data reveals a significant main effect of a single categorical factor (IV1). A follow-up LDA analysis is performed (in order to generate an easy to interpret ...
0
votes
0answers
11 views

Feature selection + cross-validation = incorrect classification score?

I'm working on a small dataset (~30) composed of many features (>100). The task consists in selecting the important positive correlated features that can distinguish the positive from the negative ...
0
votes
0answers
7 views

Error metric for cross-validation on interval-censored data?

I want to compare crossvalidated model fit (of two Bayesian models, one using a normal distribution and the other a t-distribution) on interval-censored data - data where the exact point is not known, ...
10
votes
4answers
768 views

Why splitting the data into the training and testing set is not enough

I know that in order to access the performance of the classifier I have to split the data into training/test set. But reading this: When evaluating different settings (“hyperparameters”) for ...
1
vote
1answer
26 views

Model stacking question

I'm stacking several models to improve a regression task. My question regards making final predictions when making use of stacking. My set-up is as follows. I have a train and a test data set. For ...
0
votes
0answers
16 views

Error in gbm.step: In cor(y_i, u_i) : the standard deviation is zero [on hold]

My data is called: data(Normalised.scores) I am attempting to construct a boosted regression tree using the function gbm.step() using the dismo package, where ...
0
votes
0answers
10 views

What is the limitation for k-fold cross validation in terms of samples per class?

I would like to know if it make sense to apply k-fold cross validation (e.g., k = 5 or 10), when the number of samples per class is limited, for example 4 samples for each class. Would it be better ...
0
votes
0answers
9 views

Evaluating reproducibility of more than two replicas

I'm developing a DNA sequencing-based method, which generates a lot of quantitative data. To illustrate my case, the method provides the number of times an individual mutant of a gene is observed, for ...
1
vote
0answers
14 views

How to use RFECV for feature selection and cross validation

I am still very new to machine learning and trying to figure things out myself. I am using SciKit learn and have a data set of tweets with around 20,000 features (n_features=20,000). So far I achieved ...
3
votes
2answers
45 views

Feature selection when using cross validation

I have a limited size data set of 385 entries on which I want to run multiple classifiers and compare their performance using the WEKA experimenter. The number of attributes in this data set is ...
0
votes
1answer
41 views

Time series forecasting with R

I try to forecast my web visitors on the web site for 10 future days using time series. My time series is daily. I have used an auto.arima() model. Considering ...
1
vote
1answer
24 views

Confidence interval for expected prediction error from cross-validation

I am using a support vector machine for binary classification on a sample of size 150 (75 of each class). I am using 5-fold stratified cross-validation to estimate the expected prediction error, i.e. ...
4
votes
3answers
88 views

Comparing CV Predictions across Folds for Random Forest

I have implemented a k-fold cross validation to to assess the classification performance of a Random Forest. What I want to know is: are the predicted values across folds directly comparable? For ...
1
vote
1answer
32 views

One class SVM with caret in R using cross validation

I am using one class SVM to train and predict anomalies. I would like to train the model using cross validation in an easy way as I have done with a multiclass SVM with caret in R. Now, I train the ...
0
votes
2answers
63 views

High accuracy during cross validation, low accuracy on test set

I'm currently trying to build a tennis prediction model. Unfortunately, I have some issues that I hope you could help me to handle. I have 1110 examples of matches from the year 2013, with their ...
0
votes
1answer
47 views

K-fold repeated cross validation for classification accuracy in Caret

I am new to cross-validation and I have a data-set called LDA.scores for 12 measured call-type parameters. I am trying to run a k-fold repeated cross validation with 10 folds and associated naive ...
2
votes
2answers
25 views

Double (nested,wrapper) CrossValidation - final trained model

I'm performing a study where I'm selecting kernel type and hyperparameters in an inner CV loop and an outer loop doing 10-fold CV (using SVR). The output is 10 trained models and performance measures. ...
1
vote
0answers
15 views

10-fold cross validation for forecasting time series with explanatory data ?

I saw that the question was asked some years ago here, but I wasn't satisfied with the answers so I'm asking it again. Is there some theoretical foundations about not doing k-fold cross validation ...
0
votes
0answers
17 views

Does the delta output from a k-fold cross validation indicate the estimated classification error?

I'm currently working on a logistic regression analysis and want to determine if my model validates well. I used the following R code using the "boot" package: ...
3
votes
3answers
70 views

Do we have to fix splits before 10-folds cross validation if we want to compare different algorithms?

I work with R and let's say that I have a train set and a test set. I want to test different algorithms (for example neural networks and svm). I will perform a first 10-folds cross validation on my ...
1
vote
0answers
26 views

Consistent estimate vs out-of-sample performance

When there is a cross-correlation structure in linear regression errors, the usual approach is to model the errors as an ARIMA process. It leads to a consistent estimate of the parameters of the ...
1
vote
1answer
25 views

Fail to improve recall in classification

I have a large data set with over 700,000 examples and I tried to (binary) classify the data set with Naive Bayes and Random Forest. The task was carried out in Python and Scikit-learn data The data ...
1
vote
1answer
24 views

Ratio between training error and validation error

I know that we should choose the model which minimizes validation error. But is there any meaning of ratio between training error and validation error? I was wondering if it tells something about ...
4
votes
1answer
81 views

Proof of LOOCV formula

From An Introduction to Statistical Learning by James et al., the leave-one-out cross-validation (LOOCV) estimate is defined by $$\text{CV}_{(n)} = \dfrac{1}{n}\sum\limits_{i=1}^{n}\text{MSE}_i$$ ...
0
votes
0answers
40 views

Running repeated cross-validation for multiple models using same dataset (caret package)

I'm currently using the train() function in the caret package to run 10-fold repeated cv on a random forest model. I would also like to explore other statistical and machine learning models for use ...
0
votes
1answer
20 views

Parameter stability in cross sectional data

Are there established methods to test for parameter stability for cross sectional regression? For time series regression, I am aware that Kalman filter can be used to detect parameter stability. I ...
0
votes
0answers
6 views

Does the cvglm function in boot package have arguments for mean centering and normalizing?

Does the cvglm function in boot package have arguments for mean centering and normalizing? If not, how can I run k-fold cross validation with mean centering and normalizing in R ?(especially for ...
0
votes
1answer
34 views

Differences between cross validation and bootstrapping to estimate the standard error of the AUC of a given ROC curve

I know there's been some discussion on differences between CV and bootstrapping for estimating out-of-sample prediction error of a classifier. For example, in here (Differences between cross ...
1
vote
2answers
37 views

Which data split should I use to determine cutoff point for classification?

I'm building a classification model using the caret package. I'm splitting my dataset in train and test (80/20) and training using 10-fold cross-validation repeated ...
1
vote
0answers
18 views

Which validation method to choose if the number of instances is limited?

I have a question regarding how to measure the performance of a model when the distribution of instances per class is limited. In my scenario I have five data sets $\mathcal{D}_{1}, \mathcal{D}_{2}, ...
0
votes
0answers
23 views

K-fold cross validation dealing with an interaction term

I'm working on a logistic regression analysis and have a data set that contains ~12,000 data values (~6,000 values = 1; ~6,000 values = 0). I would like to use a k-fold cross validation process to ...
0
votes
0answers
20 views

How to properly perform multiple imputations when using cross-validation procedures

I am trying to understand the association of an exposure on an outcome. In a dataset of ~600, approximately half the population does not have a measured exposure. We have predicted their exposure ...
-1
votes
1answer
28 views

Cross validation when only the regression equation is given [closed]

Is there any function in R to conduct cross validation when you only know the regression equation?
2
votes
1answer
61 views

Do I cross-validate my entire dataset, even the validation and test set?

I have the following dataset where binary_peak is a binary response variable and I have (not shown) 9 explanatory variables (also binary). ...
0
votes
1answer
28 views

Data validation

What does data validation mean in a research context? For example I am doing a linkage study, linking developing congenital anomalies with taking antibiotics in pregnancy. How can I validate the ...
0
votes
0answers
16 views

developing a prediction model for HIV outcome in cox regression using cross validation/GCV

During the application of cross-validation in sufficient large dataset (say 6000), is there a recommended ratio to split the data in to learning/training and testing/validation data set? I have seen ...
0
votes
2answers
37 views

Leave one out cross validation for neural network perfomance

When using leave one out cross validation in neural network, do I have to fix the epoch number for each training model? The test results of these models are averaged to show performance. So can I ...
2
votes
1answer
74 views

How to set the dictionary for text analysis using neural networks

I want to use a neural network to do text analysis. If I use a large dictionary, then it will contain all the words in training and test set, but the size of the dictionary is too large which will ...
1
vote
2answers
96 views

How to cross validate stepwise logistic regression?

I have a conceptual problem understanding how to cross validate stepwise logistic regression. Every time the training set is divided it is very likely that different features are chosen based on the ...
0
votes
0answers
59 views

Train neural network for forecasting

I am trying to use time series neural network to predict future values. I have time series data from 2010-2014 and I need to predict the values from 2015-2020 using time series neural network. I am ...
1
vote
0answers
33 views

too many ties in knn? how to solve this problem

I use the knn model to train my data and then eliminate accuracy via cross-validation, but when i use the following code, I get the error: Error in knn3Train(train = c(1680, 300, 480, 240, ...
7
votes
1answer
145 views

Does modeling with Random Forests require cross-validation?

As far as I've seen, opinions tend to differ about this. Best practice would certainly dictate using cross-validation (especially if comparing RFs with other algorithms on the same dataset). On the ...
0
votes
0answers
13 views

How does the value of random state affect the prediction accuracy in sklearn?

I was doing a split on my train and test data for the iris dataset and trying to randomize it. I have the following code. ...
0
votes
1answer
35 views

Cross validation for feature selection: still possible to overfit?

I would like to find a good pair of predictors out of about 400 available pairs. To do this I am using LOO cross validation. Since there are so many pairs available, don't I run into the issue that ...
2
votes
2answers
70 views

Why should validation error be higher than training error?

I was reading about learning curve and in a page, this curve is shown: But I think something is wrong with it. If an estimator tunes it parameters on validation set, then validation error should ...
1
vote
2answers
41 views

How to evaluate the final model after k-fold cross-validation

As this question and its answer pointed out, k-fold cross validation (CV) is used for model selection, e.g. choosing between linear regression and neural network. It's also suggested that after ...
0
votes
1answer
36 views

Machine Learning: Stratified Test-train-validation split for images with multiple classes and examples per image

I have a dataset with 300 images, each of which has a variable number of flowers. These flower examples can be any of 3 classes. My goal is to develop a prediction algorithm to classify the flower ...
1
vote
1answer
82 views
+50

standard deviation of cross-validation error

I am using cross-validation to estimate the prediction error of my model. I am using a metric M to measure this prediction error. Using 10-fold CV, I obtain the value of the metric M for each fold. ...
0
votes
2answers
52 views

R question about regression and cross-validation (different p-values for each)

I have an R question. I'm wondering why there is a difference in p-values in the original regression analysis using lm versus in the k-fold cross-validation using the DAAG package. So, first I run ...