Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
22 views

Running repeated cross-validation for multiple models using same dataset (caret package)

I'm currently using the train() function in the caret package to run 10-fold repeated cv on a random forest model. I would also like to explore other statistical and machine learning models for use ...
0
votes
1answer
18 views

Statistic test for comparing two methods? [on hold]

I don't have very good knowledge about statistical testing. Accept my apology if my question sounds stupid. I have developed two methods for competing discomfort level of communities. one is static ...
0
votes
1answer
12 views

Parameter stability in cross sectional data

Are there established methods to test for parameter stability for cross sectional regression? For time series regression, I am aware that Kalman filter can be used to detect parameter stability. I ...
0
votes
0answers
3 views

Does the cvglm function in boot package have arguments for mean centering and normalizing?

Does the cvglm function in boot package have arguments for mean centering and normalizing? If not, how can I run k-fold cross validation with mean centering and normalizing in R ?(especially for ...
0
votes
0answers
6 views

Differences between cross validation and bootstrapping to estimate the standard error of the AUC of a given ROC curve

I know there's been some discussion on differences between CV and bootstrapping for estimating out-of-sample prediction error of a classifier. For example, in here (Differences between cross ...
1
vote
2answers
26 views

Which data split should I use to determine cutoff point for classification?

I'm building a classification model using the caret package. I'm splitting my dataset in train and test (80/20) and training using 10-fold cross-validation repeated ...
1
vote
0answers
16 views

Which validation method to choose if the number of instances is limited?

I have a question regarding how to measure the performance of a model when the distribution of instances per class is limited. In my scenario I have five data sets $\mathcal{D}_{1}, \mathcal{D}_{2}, ...
0
votes
0answers
26 views

K-fold cross validation [on hold]

I'm working on a data set that contains used (value= 1; animal locations) and random locations (value = 0). I'm using logistic regression to assess non-random habitat selection. I have 6 continuous ...
0
votes
0answers
20 views

K-fold cross validation dealing with an interaction term

I'm working on a logistic regression analysis and have a data set that contains ~12,000 data values (~6,000 values = 1; ~6,000 values = 0). I would like to use a k-fold cross validation process to ...
0
votes
0answers
15 views

How to properly perform multiple imputations when using cross-validation procedures

I am trying to understand the association of an exposure on an outcome. In a dataset of ~600, approximately half the population does not have a measured exposure. We have predicted their exposure ...
-1
votes
0answers
18 views

How does the cross validation function in R work when the sample size is non divisible? [closed]

How does the cross validation function in R work when the sample size is non divisible by the number of folds?
-1
votes
1answer
25 views

Cross validation when only the regression equation is given [closed]

Is there any function in R to conduct cross validation when you only know the regression equation?
1
vote
1answer
47 views

Do I cross-validate my entire dataset, even the validation and test set?

I have the following dataset where binary_peak is a binary response variable and I have (not shown) 9 explanatory variables (also binary). ...
0
votes
1answer
26 views

Data validation

What does data validation mean in a research context? For example I am doing a linkage study, linking developing congenital anomalies with taking antibiotics in pregnancy. How can I validate the ...
0
votes
0answers
13 views

developing a prediction model for HIV outcome in cox regression using cross validation/GCV

During the application of cross-validation in sufficient large dataset (say 6000), is there a recommended ratio to split the data in to learning/training and testing/validation data set? I have seen ...
0
votes
2answers
32 views

Leave one out cross validation for neural network perfomance

When using leave one out cross validation in neural network, do I have to fix the epoch number for each training model? The test results of these models are averaged to show performance. So can I ...
2
votes
1answer
65 views

How to set the dictionary for text analysis using neural networks

I want to use a neural network to do text analysis. If I use a large dictionary, then it will contain all the words in training and test set, but the size of the dictionary is too large which will ...
1
vote
2answers
83 views

How to cross validate stepwise logistic regression?

I have a conceptual problem understanding how to cross validate stepwise logistic regression. Every time the training set is divided it is very likely that different features are chosen based on the ...
0
votes
0answers
44 views

Train neural network for forecasting

I am trying to use time series neural network to predict future values. I have time series data from 2010-2014 and I need to predict the values from 2015-2020 using time series neural network. I am ...
1
vote
0answers
24 views

too many ties in knn? how to solve this problem

I use the knn model to train my data and then eliminate accuracy via cross-validation, but when i use the following code, I get the error: Error in knn3Train(train = c(1680, 300, 480, 240, ...
6
votes
1answer
118 views

Does modeling with Random Forests require cross-validation?

As far as I've seen, opinions tend to differ about this. Best practice would certainly dictate using cross-validation (especially if comparing RFs with other algorithms on the same dataset). On the ...
0
votes
0answers
8 views

How does the value of random state affect the prediction accuracy in sklearn?

I was doing a split on my train and test data for the iris dataset and trying to randomize it. I have the following code. ...
0
votes
1answer
34 views

Cross validation for feature selection: still possible to overfit?

I would like to find a good pair of predictors out of about 400 available pairs. To do this I am using LOO cross validation. Since there are so many pairs available, don't I run into the issue that ...
2
votes
2answers
55 views

Why should validation error be higher than training error?

I was reading about learning curve and in a page, this curve is shown: But I think something is wrong with it. If an estimator tunes it parameters on validation set, then validation error should ...
0
votes
0answers
12 views

Comparing parametric and nonparametric models

I would like to compare some parametric and nonparametric methods. I have a small real data set (n=50), Could I use k-fold cross validation and then use some statistical distance measure like ...
1
vote
2answers
27 views

How to evaluate the final model after k-fold cross-validation

As this question and its answer pointed out, k-fold cross validation (CV) is used for model selection, e.g. choosing between linear regression and neural network. It's also suggested that after ...
0
votes
1answer
28 views

Machine Learning: Stratified Test-train-validation split for images with multiple classes and examples per image

I have a dataset with 300 images, each of which has a variable number of flowers. These flower examples can be any of 3 classes. My goal is to develop a prediction algorithm to classify the flower ...
1
vote
0answers
23 views

standard deviation of cross-validation error

I am using cross-validation to estimate the prediction error of my model. Using 10-fold CV, I obtain a bunch of metrics, including for example the MSE: ...
0
votes
2answers
50 views

R question about regression and cross-validation (different p-values for each)

I have an R question. I'm wondering why there is a difference in p-values in the original regression analysis using lm versus in the k-fold cross-validation using the DAAG package. So, first I run ...
0
votes
0answers
20 views

Resampling for unbalanced data in cross-validation

Resampling the data prior to classification is one of the techniques dealing with unbalanced dataset. I then consider down-sampling and ...
0
votes
0answers
20 views

CVlm comparison of 2 and 3-factor multiple linear regression models by cross-validation

This is a follow-on question (which I have addressed sufficiently for my requirements) from: Overfitting of Regression with Robust Variances? I am struggling to get my head around how to interpret ...
0
votes
0answers
24 views

Classification Model on Single Feature?

this is my first time using StackExchange so forgive me if I commit any faux paus with this question, and it has only been a few months since I first started learning machine learning. In my ...
0
votes
0answers
7 views

Mean accuracy confidence intervals with “corrected resampled t-test statistic”

I'm interested in calculating the expected prediction accuracy of my machine learning algorithm. My datasets only contain a small number of sample, so to do this, I repeat k-fold cross validation ...
1
vote
0answers
16 views

Interpretation of cross validation results when comparing models

I'm trying to solve a bio-medical image segmentation problem using a binary classifier and then a spatial smoothing (assuming continuous regions). I have: Training set of 10 3D scans, a total of ~30 ...
0
votes
0answers
15 views

K-fold cross validation and model containing interaction terms

I'm working on mixed effects analysis in R and want to complete a k-fold cross-validation. I ran the following model: ...
0
votes
0answers
18 views

What does it mean if the ROC scores are quite different when using the Stratified K fold with and without shuffling?

I'm currently building a random forest classification and trying to measure the model performance by the [mean ROC area]. With the same data set: When I use cross_validation.StratifiedKFold(y, ...
0
votes
0answers
20 views

How to maximize prediction for positive values (or negative values) instead of Accuracy using train function in R

I want to select and assess (using cross validation) several models in order to predict a dichotomous variable using train function in ...
0
votes
0answers
31 views

Sampling, feature selection and preprocessing in cross validation

To brief my question, I want to clarify the order of parameter tuning and the correctness of the flow in my scheme. In my classification scheme, there are several steps including: SMOTE (Synthetic ...
0
votes
1answer
37 views

Using partial AUC as Caret metric for cross-validation?

I'm evaluating a grid of tuning parameters using Caret with metric="ROC" for cross-validation. Is there any simple way to use as metric the area under the curve for an specified interval of the ROC ...
0
votes
1answer
22 views

Three way splitting and difference between CV AUC and testing AUC

I have 2000 observations in a dataset with features and a binary-class outcome. I split the dataset into two sets for split sample validation. I use 80% to train the model and internal perform Cross ...
0
votes
0answers
14 views

cross validation for cart

When a dataset is given and it is divided into N parts, training a Cart on N-1 parts and testing it on the remaining part (and doing that N times, i.e. for each possible leave-out), one ends up with N ...
0
votes
0answers
37 views

How is bootstrapping used for machine learning?

How does one use bootstrapping in a machine learning context? My typical data analysis pipeline is Split data into 10 folds Train classifier with 9 folds Test classifier with remaining fold Repeat ...
0
votes
0answers
8 views

cross validation for non parametric clustering methods: dimensionality reduction possible?

I do have about 100 data points gathered during a DoE experiment. The response variable was the settling velocity distribution depending on 10 factors. I analysed the 10 % percentile of the ...
0
votes
1answer
19 views

Cross-validating the tbats/bats function in forecast

Is there a way to cross validate the tbats/bats function in the forecast package in R? I have been trying to get CV weighted parameters which then I can pass to a function for revised estimates. ...
0
votes
0answers
5 views

Estimate variance of an arbitrary estimator using cross validation

Ron Kohavi's paper "A Study of Cross-Validation and Boostrap for Accuracy Estimation and Model Selection" explains very well how to compute the variance of the estimated accuracy when using CV (or ...
0
votes
0answers
22 views

cross validation

I am doing a study in a city, in which I have selected the Town Covers and Town composed of 13 Union Councils. I have selected all 13 unions and taken the proportion. Union Councils (UCs) are ...
1
vote
1answer
17 views

Techniques that use the addition of noise to training data

I was curious if there is a class of techniques that uses addition of noise to training data to help prevent overfitting of data. Any references would be appreciated. Thanks.
0
votes
0answers
16 views

Significance of Pattern in Out of Sample error

I am very uncertain about much of what I have done, and it is very possible I have made a significant error, due to misunderstanding, that ruins my result, so if this is the case I would greatly ...
0
votes
0answers
3 views

MCCV + Bootstrap with R?

I have a matrix of 111 observations and 1196 numeric variables. Observations consist in Diabetic / NON Diabetic persons. I want to apply Random forest or SVM as classifiers, but before I need to know ...
0
votes
1answer
33 views

What is the use of splitting dataset into training/test prior to cross-validation?

I've occasionally seen people advocate splitting the full dataset into training/test (typically a 70/30 or 80/20 split) and then running CV on the training set. I don't fully understand the point of ...