Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
9 views

cross validation for cart

When a dataset is given and it is divided into N parts, training a Cart on N-1 parts and testing it on the remaining part (and doing that N times, i.e. for each possible leave-out), one ends up with N ...
0
votes
0answers
33 views

How is bootstrapping used for machine learning?

How does one use bootstrapping in a machine learning context? My typical data analysis pipeline is Split data into 10 folds Train classifier with 9 folds Test classifier with remaining fold Repeat ...
0
votes
0answers
7 views

cross validation for non parametric clustering methods: dimensionality reduction possible?

I do have about 100 data points gathered during a DoE experiment. The response variable was the settling velocity distribution depending on 10 factors. I analysed the 10 % percentile of the ...
0
votes
1answer
13 views

Cross-validating the tbats/bats function in forecast

Is there a way to cross validate the tbats/bats function in the forecast package in R? I have been trying to get CV weighted parameters which then I can pass to a function for revised estimates. ...
0
votes
0answers
5 views

Estimate variance of an arbitrary estimator using cross validation

Ron Kohavi's paper "A Study of Cross-Validation and Boostrap for Accuracy Estimation and Model Selection" explains very well how to compute the variance of the estimated accuracy when using CV (or ...
0
votes
0answers
22 views

cross validation

I am doing a study in a city, in which I have selected the Town Covers and Town composed of 13 Union Councils. I have selected all 13 unions and taken the proportion. Union Councils (UCs) are ...
1
vote
1answer
15 views

Techniques that use the addition of noise to training data

I was curious if there is a class of techniques that uses addition of noise to training data to help prevent overfitting of data. Any references would be appreciated. Thanks.
0
votes
0answers
11 views

Significance of Pattern in Out of Sample error

I am very uncertain about much of what I have done, and it is very possible I have made a significant error, due to misunderstanding, that ruins my result, so if this is the case I would greatly ...
0
votes
0answers
2 views

MCCV + Bootstrap with R?

I have a matrix of 111 observations and 1196 numeric variables. Observations consist in Diabetic / NON Diabetic persons. I want to apply Random forest or SVM as classifiers, but before I need to know ...
0
votes
1answer
27 views

What is the use of splitting dataset into training/test prior to cross-validation?

I've occasionally seen people advocate splitting the full dataset into training/test (typically a 70/30 or 80/20 split) and then running CV on the training set. I don't fully understand the point of ...
2
votes
0answers
15 views

What is “Multi-Scale” Grid Search in SVR Cross Validation

I am trying to implement an algo that uses epsilon-SVR with a histogram intersection kernel. The slack variable C and the error intensive margin e needs to be optimised. The algo uses an exhaustive ...
0
votes
0answers
10 views

User weight specification in Meta-analysis using STATA

I want to perform Meta-analysis in STATA using User define weight option "wgt()", i.e., metan coef, fixed wgt(weight-variable). However, the model is not running, stata says, " too few variables ...
4
votes
1answer
140 views

how big to make k for cross validation

Is there a limit to how big you should make K for k-fold cross validation? I understand as K gets bigger performing the CV will take longer, but aside from that, is there any reason not to make K = n? ...
0
votes
0answers
19 views

Nested Cross validation vs Ordinary CV

Usually nested cross validation procedure is used when the tuning parameters of the model are estimated simultaneously to the model assessment. According to the theory, the ordinary CV is not suitable ...
0
votes
0answers
20 views

How to read K-Fold Cross Validation results?

If I have two models to be validated, how I could figure which model is the best? Is it the one who has bigger score, or the smaller one? Any reference for in-depth explanation and example for k-fold ...
2
votes
1answer
58 views

30% difference on accuracy between cross-validation and testing with a test set in weka? is it normal?

I'm new with weka and I have a problem with my text classification project using it. I have a train dataset with 1000 instances and one of 200 for testing. The problem is that when I try to test the ...
-1
votes
1answer
39 views

Prediction score in glmnet package r

Having the following code from glmnet package: ...
0
votes
0answers
23 views

Nested Cross Validation

I am training a linear classifier using SVM. I used nested cross validation. Inner loop is used to estimate the best C (cost parameter) and out loop uses the best C to train and test the classifier. ...
2
votes
0answers
37 views

Should I make decisions based on micro-averaged or macro-averaged evaluation measures?

I ran a 10-fold cross validation on different binary classification algortihms, with the same dataset, and recieved both Micro- and Macro averaged results. It should be mentioned that this was a ...
1
vote
2answers
17 views

Should you create a word vector before cross validation?

We are doing a lot of experiments in my research group with text data, and what usually happens is that a corpus will be transformed into instances with features as bag of word or n-gram features. We ...
1
vote
1answer
33 views

How can training and testing error comparisons be indicative of overfitting?

In my research group we are discussing if it is possible to say a model has overfitting just by comparing the two errors, without knowing anything more about the experiment. ps: I am personally ...
0
votes
0answers
20 views

How to make the right CV test?

I am going to indicate a really confused question for me in my project, I didn't learn machine learning before but I am hardworking on it, the question might be long and naive, and thanks for reading ...
0
votes
1answer
20 views

Principal component/Partial least-squares regression: can we use test data to calculate the factors?

I would like to make a PC/PLS regression and assess the resulting model's predictive power. The strategy is the classical splitting into training/validation/test sets, and using training/validation ...
1
vote
1answer
71 views

Gaussian process regression: leave-one-out prediction

According to Dubrule's Cross validation of kriging in a unique neighborhood, it is possible to compute leave-one-out the gaussian process prediction $\hat{Y}_{-i}(x_i)$ at a point $x_i$ from the ...
0
votes
0answers
30 views

Balancing random forest via cross validation. Difference between sample weight and cutoffs?

My random forest model of a simple binary target (0, 1) and is producing unbalanced results. i.e many more false positives than there are false negatives. In addition, '1' is a low percentage class, ...
2
votes
1answer
80 views

Cross validating lasso regression in R

The R function cv.glm (library: boot) calculates the estimated K-fold cross-validation prediction error for generalized linear models and returns delta. Does it make sense to use this function for a ...
3
votes
1answer
56 views

what is difference between t-test and 10-fold validation?

one of famous significant test is t-test which can conclude is our result by chance or not using previous results and their variance. Also when we use 10-fold validation we break our dataset to train ...
0
votes
0answers
22 views

Using an out-of-sample set to choose logistic regression threshold?

I am fitting a logistic regression model to 5000 examples and using the model to predict a binary dependent variable. After fitting the model coefficients, I want to binarize/threshold the model ...
0
votes
1answer
89 views

How to minimize class weight vector of Random Forest Classifier using CV

What I'd like to do is optimize the class weights of a Random Forest Classifier (using python and the sklearn library) for multiclass classification, in which different misclassification errors have ...
0
votes
1answer
30 views

How do I combine various factors/variables to a single factor/variable

I have various products and for each product I have 5 types of cost(not just monetary cost) variables associated with it, the value of each variable for any product is a positive integer. I want to ...
0
votes
3answers
61 views

Improving quality of logistic regression estimation

I'm working on a credit scoring model (logistic regression), and I have divided my dataset (5082 obs with 580 negatives) in two samples: 75% training set and 25% test set. The result of the estimation ...
0
votes
0answers
7 views

How to select optimal value of the param if tq-CrossValidation line is broken?

I've done tq-CrossValidation procedure (t = 10, q = 10). I want to select optimal value of the param. The graph of model error and param's values is next: Usually, this graph has a one minimum. In ...
0
votes
0answers
20 views

Label permutation with cross-validation

I wanted to find out if my machine learning application is prone to overfit. I first did an actual analysis with three diffent classifiers, and then repeated the whole process a few hundred times with ...
0
votes
0answers
22 views

Error: There are no cases in the testing sample

I have a problem with ANN in SPSS, I tried to run it on my data, and it works fine for one set of my data but for another set it gives me this error One or more cases in the testing or holdout ...
0
votes
1answer
28 views

Using LOOCV, AIC for Weighted Multiple Regression Model Selection?

I am currently attempting to determine the most predictive weighted multiple linear regression model to use and am trying to figure out the best combination of variables to use in the model. My first ...
2
votes
3answers
49 views

Comparing four classifiers

I have trained and tested four different classifiers, and I would now like to compare them. The classifiers have accuracies 95, 90, 81, 75. I know that there is no unbiased estimator of the variance ...
0
votes
0answers
33 views

10-fold Cross-validation vs leave-one-out cross-validation

I'm doing nested cross-validation. I have read that leave-one-out cross-validation can be biased (don't remember why). Is it better to use 10-fold cross-validation or leave-one-out cross-validation ...
3
votes
2answers
66 views

When does it makes sense to use Cross Validation?

My understanding is that cross validation is about using different chunks of the training data to train the model and average out the error estimation so that the variance is less. For example, in ...
2
votes
0answers
37 views

Regularization parameter to generate inverse covariance matrix

My data consists of approx. 5 Million binary strings (n) and every string is 2788 characters long. My goal is to find out if position i is correlated with position j. I approximated a covariance ...
0
votes
0answers
30 views

Neural Network hypothesis evaluations

If I have a sample data divided into three blocks: training, cross-validation, and test, and we want to evaluate how a model performs, which cost function error are we taking into account? I am ...
0
votes
1answer
29 views

Citing articles as references for the Leave-One-Out Method

I wish to know which work(s) should be cited as reference(s) for the Leave-One-Out method.
0
votes
1answer
39 views

How to tune parameters through cross-validation without grid search?

There are actually lots of questions about parameter tuning through cross-validation. I have read some of them, e.g. this one. I, however, still can't understand the details of the process. Here are ...
1
vote
2answers
63 views

How to use cross-validation with regularization?

I think I understand each of these concepts (cross-validation, regularization) independently, but I'm not quite clear on how they can be put together in practice. Loosely speaking, in ...
2
votes
2answers
52 views

Implement cross validation for a prediction model

I am trying to assess the predictive performance of two competing linear regression models. $$ model 1: Y \sim X_{1} + X_{2}$$ $$ model 2: Y \sim X_{1} + X_{2} + X_{3}$$ where y is continuous. I ...
0
votes
0answers
31 views

Cross-validation and nested cross-validation

I have a dataset consisting of fMRI images. Each image is 4D (time series), i.e. x-y-z-t where t is the time. Each image belongs to one class. The dataset is as follows: ...
2
votes
0answers
20 views

Leave-one-out cross validation with bayesian networks - R

I have a dataset with 1000 rows and 10 columns and s/n values. The head of the data : ...
0
votes
0answers
12 views

How many iterations we have to perform in adaboost classification?

How many iterations we have to perform in adaboost classification? As the number of iteration increases error rate gradually reduces and sometimes classification accuracy goes upto 100% in both ...
0
votes
1answer
38 views

Statistical test for classification models

I have 3 models from which, for each model, I train a classifier and then evaluate it, currently using stratified 10-fold cross validation and then take the mean accuracy ratio of these from each ...
0
votes
0answers
12 views

Is a skill score based on mean squared error appropriate for evaluating regression models based on maximum likelihood estimation?

I would like to evaluate a generalized linear model assuming a gamma distribution of the target variable and with one to several predictors by means of cross-validation. With the observational series ...
0
votes
0answers
25 views

Tune a neural network and prevent overfitting

I'm using a neural network for the first time and I would like to know if I'm doing this right. I'm working with time series for 5 years, and in each year I have a total of 18 time series plus the ...