Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...
0
votes
0answers
19 views
Which is the better method to do forecast..1-step or h-step ahead?
I am using forecast() package in R to predict future values. I have a time series data
for approx 6-7 years.
First, I split the data into training set and test
set. Test set contained values of the ...
1
vote
0answers
16 views
Does it make sense to minimize negative log likelihood on SVC probability outputs
I'd like to run a grid search cross-validation on the probability outputs of the SVC classifier. In particular I'd like to minimize the negative log likelihood. Is this a reasonable thing to do? I'm ...
0
votes
0answers
13 views
OK to use residual sum of squares for cross-validation of binary outcome?
For an OLS model the mean squared error can be used to assess the fit of the trained model on the validation data.
What is the equivalent for a logistic regression model? Can I simply use the ...
0
votes
1answer
17 views
Confusion related to regularization parameter selection by cross validation
I can see lots of paper mentioning they selected some parameters like regularization parameter $\lambda$ by cross validation. What do they mean by that?
1
vote
0answers
12 views
Cross validation with first and second order effects
Our project involves looking at different chemical properties and developing a model which incorporates first and second order effects to predict properties of new chemicals. The data set consists of ...
-1
votes
0answers
20 views
How to perform an external validation with R
I would like to do external validation using R software. So far I have used packages like "Design" and "DAAG". However they perform internal validation rather than external one. In order to perform ...
3
votes
1answer
46 views
Hold-one-out linear regression : a shortcut?
For a series of observations $(\vec{x}_i, y_i), i = 1 \cdots N$ from the linear model $Y = \beta^T X + \epsilon$, the least squares estimate of $\beta$ is: $\hat{\beta} = (\mathbf{X}^T ...
0
votes
0answers
11 views
Group statistics based on individual Granger-Causality Results
I am analyzing how external influences affect the activity of individuals in subgroups over time. To do so I calculate Grangers-causality test for every individual within the groups. My question now ...
0
votes
0answers
8 views
Do I use regularizer when I measure validation error?
I have a cost function with a regularizer term, and I'd like to find to optimal hyperparameters for the regularizer term. So I train with different parameter, but when I measure the validation error, ...
0
votes
0answers
24 views
More suitable cross validation method for estimation method
I have a sparse dataset of graph of size about (7k*7k).I estimate some values for each not existence edge according to the information of graph. I want to validate the method(The accuracy of the ...
0
votes
0answers
31 views
lasso and cross-validation (theoretical results)
is there any theoretical result which says that use the minimum of the cross-validation as value for the lasso penalty is a good choice?
I would like something like $P(S_0 \subset \hat ...
0
votes
0answers
27 views
incorporating averaging models from AIC and still using k-fold cross validation?
Ive a county/district that Ive divided into ~300 grids that are 15km^2 in size attributed with various habitat and economic variables that have been summarized and standardized. I then have 2 types ...
4
votes
1answer
49 views
Equivalence between single sample cross-validation index and the Akaike information criterion for prediction
In "Cross-Validation Methods. Journal of mathematical psychology, Vol. 44, No. 1. (March 2000), pp. 108-132", Professor Browne pointed out that single sample cross-validation index and the Akaike ...
2
votes
1answer
37 views
How to report the results of cross-validation for comparing two models?
I want to compare the predictive power of two models. For this, I calculated the difference in some measure of predictive performance over many cross-validation replications. Now I have a distribution ...
1
vote
0answers
22 views
Conditional cross-validation sampling
When applying cross-validation to evaluate the predictive performance of a binary classification model, is it acceptable to separately sample from cases and non-cases to achieve class proportions in ...
3
votes
3answers
27 views
Utilizing cross-validation with up-sampled data
I am creating a support vector machine for extremely unbalanced data in which identifying instances of the rare class is of the highest importance. Since the data is so unbalanced, training and ...
1
vote
0answers
20 views
Non-nested model verification
My questions are two fold:
Is there a generally accepted statistic used to compare non-nested, nonlinear models with different numbers of parameters? I'm thinking RMSE, but wondering what other ...
0
votes
0answers
32 views
Repeated 100x 10-fold cross validation, what is the sample size when doing an significance test?
I iterated my 10-fold cross validation 100 times for several methods. Now I want to use a t-test to test if the results are significant. However, I'm not sure what the sample size is.
Is the sample ...
4
votes
1answer
57 views
Can I repeat cross validation with a small dataset, and/or how can I improve my cross validation confidence?
For university we need to classify 3 cancer types and give an estimation of how well our model will perform. We received a dataset with 100 samples. We split the data up into a training and test set ...
1
vote
0answers
64 views
k-fold cross validation vs k times hold-out validation
I am facing the evaluation of a genetic programming algorithm. I am using the Proben1 cancer1 dataset to evaluate the models created by this algorithm. This dataset contains 699 samples, which is ...
0
votes
0answers
41 views
Interpreting output of 10-fold CV on classification tree
Using info from Decision tree model evaluation for "training set " vs "testing set " in R , I was able to run a 10-fold cross validation on my entire dataset, using this command:
...
0
votes
0answers
23 views
Cross-validation of a SMA (standardized major axis regression) model
How to cross-validate a standardized (reduced) major axis regression (SMA) model in the Smatr package (this package has no predict function)?
I have used boot, bootstrap, DAAG, cvTools, rms and Zelig ...
4
votes
0answers
63 views
Implementation of the cross validiation
I'm attending a course in computational statistics, which should be an applied course. We study different methods, which are important in "reality". One of these topics is Cross Validation. I'm faced ...
0
votes
0answers
39 views
How to decide the order of my ARMAX-model for each component?
I'm doing time series analysis with 78888 x 8 matrix of data.
The matrix includes the response data (the series I'm interested in predicting) and exogenous data. The data is hourly sampled and ...
3
votes
1answer
60 views
Is it ok to determine early stopping using the validation set in 10-fold cross-validation?
I am working on a machine learning experiment comparing the use of multiple different neural network classifiers by applying them on a large number of datasets, using stratified 10-fold ...
1
vote
2answers
111 views
Performing model validation in stata
I need to validate a cox-regression model internally - i was thinking of either k-fold cross validation or bootstrapping methods. are either of these possible to do with a cox model? any idea how to ...
3
votes
1answer
37 views
Testing a dataset against a distribution with parameters estimated from that dataset
I am trying to figure out the best distribution to fit some data to, and I'm not sure if what I am doing is statistically correct. My data consists of 20 samples / year over 10 years. For each sample ...
3
votes
1answer
70 views
PCA before train/test split
I have a dataset for which I have multiple sets of binary labels. For each set of labels, I train a classifier, evaluating it by cross-validation.
I want to reduce dimensionality using PCA. My ...
5
votes
1answer
69 views
What's the best way to choose data for Crossvalidation on linear regression settings (PCA, PLS)
We are extracting features from EEG, which is a time dependent signal.
We have signals of 10,000 datapoints over 64 channels, and we extract 10 features per timestamp per channel, so at the end we ...
3
votes
5answers
134 views
How do you decide what your train, validation and test percentages are?
When splitting up my labeled data into training, validation and test sets, I have heard everything from 50/25/25 to 85/5/10. I am sure this depends on how you are going to use your model and how ...
1
vote
1answer
31 views
Binary classifier testing with minority positive data
I have a question about a testing methodology for a binary classifier, that I'm not entirely sure how to describe. So apologies if this has been answered many times before but I haven't had much luck ...
0
votes
0answers
39 views
CV acc mismatch the prediction
Setting the Context :
My project is in C++, I'm using OpenCV svm here
I used the function train_auto for the CV, however, I implemented my own cross-validation base on this Matlab example here (I ...
3
votes
1answer
124 views
Is it possible to compare two feature selections algorithms by cross-validations?
Assume I have two feature selection algorithms, A and B, which are developed based on SVM. I applied these two algorithms on the same dataset, a Liver Cancer dataset (400 features & 150 samples), ...
2
votes
2answers
183 views
Choosing a predictive model after k-fold cross-validation
I am wondering how to choose a predictive model after doing K-fold cross-validation.
This may be awkwardly phrased, so let me explain in more detail: I understand how K-fold cross-validation works. ...
1
vote
1answer
117 views
How does cross validation work in R's gbm package?
Can someone provide a work flow about this? For instance, suppose I am doing binary classification,
For each iteration of the algorithm:
Randomly sample k*N rows, where k is the bag.fraction, and N ...
1
vote
1answer
42 views
Inconsistency in cross-validation results
I have a set of dataset recorded from subjects as they perform some particular cognitive task. The data consists of 16 channels and a number of sample points per channel and I want to classify this ...
1
vote
1answer
85 views
Use cross validation to calculate a ridge regression parameter
I have a dataset of text messages of which I'm trying to filter out the spam from the legitimate ones. I have roughly 4600 pieces of data spread among 57 features and then their classification as spam ...
1
vote
0answers
95 views
K-fold vs. Monte Carlo Cross Validation
I am trying to learn various cross validation methods, primarily with intention to apply to supervised multivariate analysis techniques. Two I have come across are K-fold and Monte Carlo Cross ...
3
votes
1answer
97 views
Graphical representation of cross-validation errors for regression
What are some good ways of presenting/comparing cross-validated RMSE errors for regression using various models, graphically via plots? As of now, I have been presenting the quantitative results in ...
0
votes
1answer
169 views
Cross-validation accuracy interpretation -( accuracy of 100%
Here the setup:
I have 90% of the data for training and other 10% for testing.
I am doing stratified cross-validation on the 90% Tranining. It is a 10-class dataset. I am using LibSVM for that. ...
3
votes
0answers
432 views
Validation: Data splitting into training vs. test datasets
I was naively validating my binomial logit models by testing on a test dataset. I had randomly divided the available data (~2000 rows) into training (~1500) and validation (~500) datasets.
I now ...
1
vote
0answers
57 views
Which measure could tell me about what is the best predictor in survival analysis?
I have data of cancer survival, like that:
> my.data
survival stage my_class
27 3 2
221 2 1
43 3 3
...
The survival is the time ...
3
votes
1answer
50 views
Mixing User Data For Cross-Validation
I have data from 12 users, and want to perform cross-validation. Is it necessary that I create my training and test data from different users, or can all of the data be randomized, and then split into ...
2
votes
4answers
187 views
Success of a logistic regression model
Say I have a model $y=f_n(x_1,x_2,x_3)$.
Here say $y$ is categorical and binomial response. i.e. $y$ can be only 0 or 1. Data shows 87% 1 and 13% 0 values.
I fit a multinomial logit on a test ...
0
votes
0answers
17 views
Difference between LGOCV and repeated bootstrap [duplicate]
The command trainControl in the R package caret supports:
boot
boot632
cv
repeatedcv
LOOCV
LGOCV
oob
What is the difference ...
2
votes
0answers
45 views
Rule of thumb for tuning the values of the penalty parameter in SVM models
I have recently been running into computational issues in fitting a soft-margin SVM model using the e1071 package in R.
The issue is unavoidable since the problem ...
1
vote
0answers
54 views
Training with very few positives
I have a binary classification problem where the fraction of positives is very low, e.g. 20 positives in 10,000 examples (0.2%)
What is an appropriate cross validation scheme for training a ...
8
votes
4answers
649 views
Why do researchers use 10-fold cross validation instead of testing on a validation set?
I have read a lot of research papers about sentiment classification and related topics.
Most of them use 10-fold cross validation to train and test classifiers. That means that no separate ...
2
votes
1answer
101 views
What is deviance in lassoglm
I am trying to fit a lasso penalized logistic regression model to a certain data. I am using lassoglm for that in matlab. I use the following function
[B,FitInfo] = ...
0
votes
0answers
35 views
Understanding stratified CV
What is the difference between stratified CV and CV?
Wikipedia says:
In stratified k-fold cross-validation, the folds are selected so that
the mean response value is approximately equal in all ...
