Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
1answer
15 views

How to tune parameters through cross-validation without grid search?

There are actually lots of questions about parameter tuning through cross-validation. I have read some of them, e.g. this one. I, however, still can't understand the details of the process. Here are ...
0
votes
0answers
4 views

Plot cv.glmnet with norm in x axis [on hold]

I would like to make comparable plots of Ridge coefficients and CV error, having the L1 norm on the x axis. I can do that for the ridge plot by fixing xvar="norm". Is there some equivalence for ...
1
vote
2answers
51 views

How to use cross-validation with regularization?

I think I understand each of these concepts (cross-validation, regularization) independently, but I'm not quite clear on how they can be put together in practice. Loosely speaking, in ...
2
votes
2answers
36 views

Implement cross validation for a prediction model

I am trying to assess the predictive performance of two competing linear regression models. $$ model 1: Y \sim X_{1} + X_{2}$$ $$ model 2: Y \sim X_{1} + X_{2} + X_{3}$$ where y is continuous. I ...
0
votes
0answers
25 views

Cross-validation and nested cross-validation

I have a dataset consisting of fMRI images. Each image is 4D (time series), i.e. x-y-z-t where t is the time. Each image belongs to one class. The dataset is as follows: ...
1
vote
0answers
7 views

Leave-one-out cross validation with bayesian networks - R

I have a dataset with 1000 rows and 10 columns and s/n values. The head of the data : ...
1
vote
0answers
17 views

LOOCV vs. AIC for Weighted Multiple Regression Model Selection

I am currently trying to determine the most predictive weighted multiple linear regression model to use. I don't have much formal statistical training, so I would greatly appreciate any help with the ...
0
votes
0answers
9 views

How many iterations we have to perform in adaboost classification?

How many iterations we have to perform in adaboost classification? As the number of iteration increases error rate gradually reduces and sometimes classification accuracy goes upto 100% in both ...
0
votes
1answer
25 views

Statistical test for classification models

I have 3 models from which, for each model, I train a classifier and then evaluate it, currently using stratified 10-fold cross validation and then take the mean accuracy ratio of these from each ...
0
votes
0answers
9 views

Is a skill score based on mean squared error appropriate for evaluating regression models based on maximum likelihood estimation?

I would like to evaluate a generalized linear model assuming a gamma distribution of the target variable and with one to several predictors by means of cross-validation. With the observational series ...
0
votes
0answers
17 views

Tune a neural network and prevent overfitting

I'm using a neural network for the first time and I would like to know if I'm doing this right. I'm working with time series for 5 years, and in each year I have a total of 18 time series plus the ...
3
votes
0answers
27 views

Variance of binomial vs. multinomial distribution in cross-validation

Suppose we have a dataset with $N=100$ observations. We do $K$-fold cross-validation with $K=10$ and $K=100$. In the first case, the classification decisions are sampled (can I say it like this?) ...
0
votes
0answers
16 views

K-fold cross validation for a glmer model with nested data

I'm working on a data set that contains a hierarchical data structure (i.e., GPS locations nested within individual animals). I'm using a generalized linear mixed effects modeling procedure (lme4 ...
0
votes
0answers
19 views

Normalize time series data - Wikipedia article counts

I have: 3 wikipedia article access counts (weekly) (A-B-C) Ground truth data (weekly) Total wikipedia english article traffic counts (weekly) My purpose is, build a multiple linear regression ...
2
votes
1answer
82 views

How do you use test data set after Cross-validation

In some lectures and tutorials they suggest to split your data in three parts: training, validation and test. But it is not clear how test data set should be used and how this approach is better than ...
0
votes
0answers
36 views

Need help identifying which stat test to use

I have raw data. Without diving into the details, I'll try to explain the principles of what is represents; It's collected from 2 different regions. Raw data is a bunch of random time length amounts ...
0
votes
0answers
5 views

Developing cross validated regression model (nlinfit) in matab [migrated]

I am using the following code to fit a cross-validated non-linear regression model. ...
0
votes
0answers
9 views

Strange xerror when using loss matrix

I really hope someone can help me. I have built a decision tree for a very skewed dataset (2.6 pct yes, 97.4 pct no). So to get high risk and low risk groups rather than getting a tree with root node ...
0
votes
0answers
22 views

Finding a curve in a given dataset

Here we have two sets of data points - red and black as shown in the attached Figure. Please could you provide me a right direction to answer the following two questions: 1) between t1 and tn, given ...
0
votes
2answers
55 views

Data splitting and cross validation

my question is about splitting data! I used to split data into training and testing set using caret library in R ...
0
votes
1answer
10 views

Propagating RMSECV?

I have two regression models, each of which has an associated root mean squared error of cross validation (RMSECV). I would like to combine the results of the models using a weighted average to get a ...
5
votes
1answer
96 views

Does the opposite of nested cross-validation make sense?

I'm asking the question from a machine learning point of view. I have a dataset with relatively high sparsity, so if I use nested cross-validation for my feature tuning and evaluation, that is tune ...
3
votes
2answers
285 views

My Test accuracy is pretty bad compared to cross-validation accuracy

I did a Multi-class document classification. I divided the original data set (18,8334 documents as a list of strings where each element of list is a document string.) into 70% training and 30% test. ...
1
vote
2answers
49 views

Nested cross-validation - how is it different from model selection via kfold CV on the training set?

I often see people talking about 5x2 cross-validation as a special case of nested cross validation. I assume the first number (here: 5) refers to the number of folds in the inner loop and the second ...
0
votes
0answers
20 views

Post-hoc analysis of variable selection

I am using support vector machines & 10-fold cross-validation for a binary classification task. For feature selection, I use the t-test. After doing the classification, I'd like to do a post-hoc ...
0
votes
0answers
8 views

Different behaviour for local regression function [migrated]

I'm fairly new to R and am trying to build a function similar to this. I have hacked the code with the aim of running locpoly to fit a local polynomial with an ...
0
votes
0answers
24 views

Why can I use the posterior probability of a classifier as a new classifier?

I have read that, when doing discriminant analysis, you can use the posterior probability you obtain using your classifier as a new fine-tuned classifier. Can anyone talk me through the rationale of ...
0
votes
1answer
31 views

Non-linear model fitting

I would like to fit a non-linear model that looks like the following: $V(g)=a*A(g)/(b*B(g)+c*C(g))$, where $g$ represents a gene, $a$, $b$ and $c$ are coefficients of $A(g)$, $B(g)$, $C(g)$, which ...
0
votes
0answers
19 views

Reasons for GLM ('identity') performing better than GLM ('gamma') for predicting a gamma distributed variable?

I am investigating different methods for fitting my target variable (observed wind speed: positive, real, with small values being most probable) using generalized linear modeling (GLM) and - in a ...
1
vote
2answers
49 views

How to estimate variance of classifier on test set?

I have a binary classification task for which I want to compare two different classification methods as well as hyper-parameters for each. I have used k-fold cross-validation (k = 5) to obtain k ...
0
votes
1answer
41 views

SVM parameter tuning for unbalanced classes (with class weights)

I am trying to run an SVM on an imbalanced dataset (0-90%, 1-10%) using the e1071 package, with the radial kernel. I am using cross-validation to select the best gamma and cost. Additionally, I want ...
0
votes
2answers
34 views

How to find the best coefficient vector using cross-validation

So basically my dataset is divided into 5 train and 5 test folds. This is how I did in scikit: ...
1
vote
1answer
51 views

Training an SVM and performing cross validation

I am training an SVM and I have 40k Negative Samples and 17k Positive samples. What I did is that I have divided my samples into training and testing subsets. In order to train the SVM I have used ...
0
votes
0answers
28 views

how to make or prepare range file in svm-scale in libsvm using matlab

Respected all, I am using LIBSVM, for scaling the input data svm scale function is used. The syntax is 'svm-scale -l -1 -u 1 -s range train > train.scale' or svm-scale -s scaling_parameters ...
0
votes
1answer
32 views

Is cross validation just varying the size of training and validation sets or is there more to it?

Starting from Linear Regression, we always divide our dataset into training and test. Once we come to cross validation, this is now training and validation sets. In validation approach, we divide ...
0
votes
0answers
27 views

k-fold on dataset

I have been doing a specific check of k-fold technique to see the difference using different number of folds and the corresponding result on the score obtained. To perform this test I have made ...
6
votes
2answers
104 views

Back-testing or cross-validating when the model-building process was interactive

I have some predictive models whose performance I would like to back-test (i.e., take my dataset, "rewind" it to a previous point in time, and see how the model would have performed prospectively). ...
1
vote
0answers
27 views

Comparing predictors based on ROC AUC and cross-validation error

I am analysing how well some continuous variables (e.g. weight, height) predict the occurrence of a given disease after surgery. I have computed the area under the curve of the receiver-operator ...
1
vote
1answer
45 views

Cross validation with test data set

I am a bit confused about the application of cross-validation. So, if I have a big data set, I will split my data into test and training data and and perform validation on the test data. But if I have ...
0
votes
0answers
19 views

Testing accuracy keeps on increasing. No signs of overfitting

I am trying to do some cross-validating in R. As I am increasing the cost parameter in LiblineaR, the testing accuracy should increase initially, but start to decrease. However, as my cost parameter ...
0
votes
0answers
22 views

Including Feature Selection in Cross Validation - Application to Bag of Words

I am working on a prediction problem where I was given a 6,000 record dataset with the value of the dependent variable included ("train"), and a 2,000 record dataset with the same independent ...
4
votes
1answer
76 views

May I use the whole dataset to prove the existence of a confounding variable in a machine learning framework if I don't use the labels?

I have a certain dataset that I am analyzing with machine learning techniques. I believe there is a certain variable (not used for training or testing the classifiers but is still known) that has an ...
1
vote
1answer
41 views

out-of-bag error estimate for Boosted Trees

In Random Forest, each tree is grown in parallel on a unique boostrap sample of the data. Because each boostrap sample is expected to contain about 63% of unique observations, this lefts roughly 37% ...
2
votes
1answer
39 views

What is the chance level accuracy in unbalanced classification problems?

Suppose one has a balanced classification problem (50% of 0's and 50% of 1's). In such a case, the so called chance-level accuracy of classifier would be 50%. What is the chance-level accuracy if the ...
0
votes
0answers
16 views

What does “10-fold cross validation” mean when only a training set is provided? [duplicate]

Recently I encounter "10 cross validation" several times in literature reading as well as using a data mining tool "Weka". I checked wikipedia, it states "One round of cross-validation involves ...
-1
votes
1answer
32 views

Somebody explain Training, Testing and Validation Test of Artificial Neural Network [duplicate]

What is the procedure of Training, Testing and Validation Test? Explain it thoroughly. Or give some link for related articles
5
votes
1answer
53 views

What if high validation accuracy but low test accuracy in research?

I have a specific question about validation in machine learning research. As we know, the machine learning regime asks researchers to train their models on the training data, choose from candidate ...
0
votes
0answers
23 views

Overfitting in the validation set

When running an algorithm for training a system it is common to consider a lot of models and using the validation set for selecting one of them. In my case I am running a mini-batch gradient descent ...
1
vote
0answers
24 views

Cross validation before or after stepwise modeling [duplicate]

I have a dataset of 1931 observations and I intend to predict a binary outcome out of that. There is a list of 128 predictors (both binary and continuous). First I ran logistic regression modeling ...
0
votes
1answer
47 views

LOOCV $R^2$ higher than regular $R^2$ in RF

I am working with RF and the caret package, and I am having a confusion because sometimes the LOOCV $R^2$ is higher than the regular $R^2$. Is it right? How can I interpret this? Here an example ...