Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
8 views

Binary classification with too few positive samples

I met a problem of doing a binary classification with quite few positive samples. For example: Binary classification with either labelled 1 samples (positive) or labelled 0 samples (negative ...
0
votes
0answers
7 views

Is there an effect size for Kappa's?

I am staring a project on comparing standard ways of creating a classifier with some heuristic methods. The heuristic methods should result in a faster training for the classifier but should result in ...
0
votes
0answers
11 views

Repeated CV vs. CV

Suppose a data set contains the numbers $1-100$ (i.e. $\{1, \dots, 100 \}$). In $10$-fold cross validation, the data set is divided into $10$ subsets with one used as the validation data set. For ...
0
votes
0answers
19 views

Constant RMSE margin between training and teseting set

I have a large number of independent datasets of varying size but same feature meaning. Features and outcome are both binary. I am trying to fit logistic regression to the data. I estimate ...
5
votes
2answers
102 views

AIC, BIC and GCV: what is best for making decision in penalized regression methods?

My general understanding is AIC deals with the trade-off between the goodness of fit of the model and the complexity of the model. $AIC =2k -2ln(L)$ $k$ = number of parameters in the model $L$ = ...
1
vote
0answers
9 views

CV for LASSO tuning parameter using LARS

If I use the LARS algorithm to fit the LASSO path, is it sufficient to cross-validate using the values of $\lambda$ at each step in LARS or is it better to use a finer grid of $\lambda$ values? I ...
0
votes
0answers
12 views

Getting the rule from cross validation

I've got a question. Let's say I have a medical data representing 2 classes of patients (healthy and unhealthy) and some number of predictors which characterize these patients. Choosing different ...
1
vote
0answers
30 views

What are appropriate validation methods for a Bayesian network model with low sample size?

I am currently using a Bayesian network model with 20 variables and 210 data points, with 15 locations measured at 14 different time points each. There are also some restrictions on what types of ...
1
vote
0answers
22 views

Is hold-out validation a better approximation of “getting new data” than k-fold CV?

I've been rethinking an answer I gave to a question a couple weeks ago Hold-out cross-validation produces a single test set that can be used repeatedly for demonstration. We all seem to agree that ...
0
votes
1answer
30 views

Interpretation of Output

Here is the R code that produced the output below: ...
1
vote
1answer
45 views

Is cross validation needed?

Suppose we have training data set and a test data set. The outcome variable is binary. Is it usually necessary to split the training data set so that there is a cross validation data set? Or can you ...
2
votes
2answers
32 views

The size of the sample for split validation

At this moment I have a dataset with 4000 samples (50% positive and 50% negative). Normally I would do cross validation for this approach, however besides normal data mining techniques I am also ...
0
votes
0answers
6 views

Liblinear logisitic regression with L2 regularization for classification

I am trying to use the liblinear library for logistic regression with L2 regularization. However, I am finding some issues with it. For eg when choosing the cost parameter, I chose the C parameter to ...
1
vote
0answers
29 views

Using Leave-One-Out Cross Validation with LARS

I have a kind of obscure question about using the Least Angle Regression (LARS) algorithm for variable selection. If I'm understanding it right, my professor formulates LARS as such: $$\mathbb{min}\ ...
7
votes
3answers
314 views

How is cross validation different from data snooping?

I just finished "An Introduction to Statistical Learning". I wondered whether using cross-validation to find the best tuning parameters for various machine learning techniques is different from data ...
0
votes
0answers
22 views

Are there any well-understood circumstances when we should expect cross-validation to be indifferent to $\alpha$ in elastic net tuning?

I am fitting a lot of elastic net models, simultaneously tuning $\lambda$ and $\alpha$. I am often coming to the following conclusions: Cross-validation error is much more sensitive to changes in ...
1
vote
0answers
6 views

Random seed choice changes qualitative results of elastic net parameter tuning

Below is a function I wrote to try and tune the $\lambda$ and $\alpha$ elastic net GLM implemented with cv.glmnet. I've noticed that the qualitative outcome (in ...
3
votes
1answer
66 views

Variance-covariance matrix for ridge regression with stochastic $\lambda$

In ridge regression with design matrix $X$, outcomes $y$, fixed regularization parameter $\lambda$, and errors $\epsilon\sim\mathcal{N}(0, \sigma^2I)$, the computations for the ridge regression ...
0
votes
0answers
30 views

Training and testing on Unbalanced Data Set

I used SMOTE algorithm in R for class balancing. My data size has 13000 rows, I had 7% minority class in my sample now I used SMOTE( Synthetic Minority Oversampling Technique) for class balancing such ...
0
votes
0answers
15 views

visualize the naive bayes with k-fold cv

I have done the classification using naive Bayes as a classifier, and applied 10-fold CV.I know that I can get the mean and variance of the result. However, how can I plot the classifier performance? ...
0
votes
0answers
42 views

using Root Mean Squared Error (RMSE) to compare models with different sample size

I'm using k-fold cross-validation to compare different models. I splitted my dataset in 6 chunks and used 4 random chunks as training set and the remaining 2 as a test set. Now I fitted n-different ...
0
votes
0answers
37 views

BRT analysis using count data

I have some problems with my BRT analysis. Introduction to the data: The dependent variable is count data of a specific palm species in SA, and the predictors consists of nine various kinds of ...
0
votes
1answer
27 views

Dividing up Training Data into Test Set

Suppose we have a training data set. We want to learn some hypothesis using some algorithm. Would we divide up the training set differently if we used, for example, logistic regression as opposed to ...
2
votes
0answers
26 views

Calculate prediction errors of binary model: What's a good way?

I am totally new to statistics, so this may be obvious, but I don't get it. Basicly, I fit a special kind of tree-model to a subset of data (one half), and now I want to cross-validate my model ...
1
vote
0answers
23 views

Cross-Validation vs. AICc for LASSO

I was working on a research project in which I try to estimate the the individual contribution of a group of regional political leaders to local economic growth. The major challenge is that there is ...
0
votes
0answers
31 views

R cv.glm returns NaN for stepwise-generated regression model

I'm trying to run K-fold cross-validation on a multiple regression model that was generated via the step function in R. However, the call to ...
5
votes
0answers
86 views

Minimum sample size for a dichotomous outcome

I have two questions. I am running an experiment where I am interested in determining the sample size required for a certain CI and error, where values range between $<1$ and $>-1$. However, I ...
0
votes
0answers
22 views

Question Regarding Nested-Cross Validation

I am slightly confused of the setup for tuning hyperparameters under gridsearch. Below is Python-style pseudo code for my understanding. ...
0
votes
0answers
12 views

Validating a multivariate categorical model

I assume that my population is a sample of an unknown multivariate categorical distribution $\mathbf{X} = (X_1, X_2, \ldots, X_k)$. From this population, a sample $\mathbf{X^*}$ is available, I assume ...
5
votes
3answers
202 views

K-fold or hold-out cross validation for ridge regression using R

I am working on cross-validation of prediction of my data with 200 subjects and 1000 variables. I am interested ridge regression as number of variables (I want to use) is greater than number of ...
0
votes
0answers
13 views

Neural nets method? Am i doing something wrong?

To set the scene - i am using neural nets in SAS program i have ~ 1000 points variables have been selected prior to this on a combination of human knowledge and variable recombination (PCA etc) I am ...
2
votes
2answers
80 views

Model Tuning and Model Evaluation in Machine Learning

Despite my readings (on stack 1, 2, or in literature (Cawley, 2010; Japkowicz, 2011)), I don't find a clear procedure for tuning and evaluating a model in a classification task. I want to perform a ...
9
votes
4answers
456 views

Hold-out Validation vs K-Fold Validation?

To me, it seems that Hold-out validation is useless. That is, splitting the original dataset into two-parts (training and testing) and using the testing score as a generalization measure, is somewhat ...
1
vote
0answers
23 views

Unstable models, repeated crossvalidation, feature selection

I'm still trying to classify few (about 200) samples in a high dimensional feature space (dim=19) into 3 (very unbalanced) classes. I use an implementation of Least Squares SVM with one vs one coding ...
0
votes
2answers
31 views

cross validation in ridge regression for classification. regularization issue

I perform ridge regression for classification. To find regularization parameter I do K-fold cross-validation with classification accuracy as a measure. This gives me some $\lambda$, which I then use ...
0
votes
0answers
13 views

Cross validation to test the performance of two spatial interpolation methods

I have 14 weather stations' temperature data for the period between 2010–2013. I need to evaluate the performance of two spatial interpolation methods. I suggest to select 10 days from this period of ...
2
votes
0answers
29 views

Cross validating quantile regression

I applied quantile regression on some data and did it for tau = 0.25, 0.5, 0.75. After i got the estimates of each model, i did some cross validation on my hold out data. When i used the estimates for ...
0
votes
0answers
21 views

Bias/Variance Trade-off in Cross-Validation

As I understand it, in K-fold cross validation, the bias gets smaller as K gets larger but the variance increases too. I'm having problems in intuitively understanding this concept from the variance ...
1
vote
1answer
43 views

Overfitting in Genetic Programming

I've recently started experimenting with Genetic Programming as an optimization tool. I'm still a little confused as to how to reduce overfitting in this framework. A couple of techniques I've read ...
1
vote
1answer
72 views

Random forest cross validation for feature selection, imbalanced datasets

I have an 5297X26 imbalanced dataset, the class1 has 588 samples and class2 has 4709 samples. I used the following code to perform random forest: ...
0
votes
0answers
11 views

Cross-validation of panel data

When I perform k-fold cross validation, the observations are selected randomly into each fold. This works fine for cross-sectional data. However, when I have panel data (and fit with hierarchical ...
2
votes
2answers
56 views

Use of nested cross-validation

Scikit Learn's page on Model Selection mentions the use of nested cross-validation: ...
0
votes
0answers
23 views

Cross validation and accuracy calculation in lib-linear

I have two questions related to cross validation in LIBLINEAR I have 1000 documents from which i take 300 documents for training and rest 700 for classification . I train 300 documents with ...
1
vote
0answers
35 views

How many digits make sense for cross-validation?

I am currently developing a mathematical symbol classifier (see http://write-math.com) for my bachelors thesis in computer science. To evaluate what type of classifiation / features / parameters work, ...
2
votes
2answers
87 views

How to find optimal values for the tuning parameters in boosting trees ?

I realise that there are 3 tuning parameters in the boosting trees model, i.e. the number of trees (number of iterations) shrinkage parameter number of splits (size of each constituent trees) My ...
13
votes
2answers
342 views

How do I know which method of cross validation is best?

I am trying to figure out which cross validation method is best for my situation. The following data are just an example for working through the issue (in R), but my real ...
0
votes
1answer
59 views

Leave-one-out cross validation

I am working about elaborate a simple linear regression so I need to evaluate many models. I am asking how I can use leave-one-out cross validation to validate a simple linear regression.
1
vote
0answers
50 views

Leave one out cross validation for rare events logistic regression in SPSS

I am writing thesis about predicting which corporate takeover targets attract a competing bidder. I first estimated a model using a logistic regression. The sample consists of 1350 single-bidder ...
2
votes
1answer
65 views

SVM parameter selection and model testing with cross-validation

I've read: Model selection and cross-validation: The right way Crossvalidation and/or testdata. Always use both or can one exclude the other? but I still don't get it. My problem is to construct ...
0
votes
1answer
19 views

Literature for Cross Validation on Sparse Data?

I've read a lot about Cross Validation to estimate prediction error, specifically for selecting the number of components in a PCA model (I'm not doing SVD/PCA, but it's very similar), but I can't find ...