Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
10 views

Parameter estimation of gaussian function kernel using cross-validation

I need to estimate (using cross-validation), the parameters $\sigma$ and $\lambda$ of the Gaussian kernel: $K_G(x,y) = \sigma^2 \exp{(-\frac{1}{2\lambda^2}\sum_{i,j}(x_{ij}-y_{ij})^2})$ where $x$ ...
0
votes
2answers
22 views

Cross Vaidation in Factor Analysis

I am trying to conduct an EFA with a sample size of 150 respondents. I would also like to use cross-validation but my professor says that the sample is not big enough for that. Is that true?
2
votes
1answer
59 views

Why should all Cross-Validation results be higher than the result on the test dataset?

Sorry, I'm not an expert and my question could be fundamentally wrong. I've read this interesting question because I also was wondering whether to train the model again after cross-validation. Now, ...
0
votes
1answer
28 views

How do you measure the accuracy of an inference hypothesis/procedure?

Take inference to mean reasoning/predicting the value of a hidden/laten variable $Z$ given some evidence/data $X$. For example, maybe you are trying to find out if your patient has Cancer (Z = 1 if he ...
0
votes
1answer
16 views

Feature Selection using (low) MCC

I have approximately 1200 input parameters that I am trying to whittle down with the following rough process: 1) Fit rbf SVM with n = 1200 parameters and calculate Matthews Correlation ...
0
votes
1answer
20 views

Predictive modeling techniques for in-sample rather than out-of-sample prediction?

Is it appropriate to apply predictive modeling variable selection and shrinkage techniques (for example, ridge regression or lasso) for in-sample prediction rather than out-of-sample prediction? ...
2
votes
2answers
83 views

About cross-validation for machine learning

Assume I have 1000 samples of data. I split the data randomly into training and test sets of size 800 and 200, respectively. Now, I train a classifier using the training set, and then evaluate the ...
3
votes
2answers
51 views

Assigning even partitions for Cross-Validation

This is a very basic question about cross-validation. Say that I have a sample size of 2901(or any difficult to divide number). How do I split this up into equal partitions (other than n=1)? And how ...
2
votes
0answers
15 views

Use of Deviance Residuals for Leave-One-Out Cross Validation

I am a newbie to stats and having some difficulties understanding how to use deviance residuals for leave-one-out cross validation for a logistic regression model. The problem that I am trying to ...
0
votes
1answer
17 views

Leave-one-subject-out cross validation in Caret

Hi Dear Colleagues, I wonder how to correctly setup a leave-one-subject-out cross validation (LOSO) for train() function in caret. Here is my example code: ...
0
votes
0answers
31 views

10 fold cross validation model in weka

Trying to build a specific Neural Network arcitecture and testing it using 10 fold cross validation of a dataset. Now building the model is a tedious job and Weka expects me to make it 10 times for ...
0
votes
0answers
12 views

Binary classification with too few positive samples

I met a problem of doing a binary classification with quite few positive samples. For example: Binary classification with either labelled 1 samples (positive) or labelled 0 samples (negative ...
0
votes
0answers
24 views
+50

Is there an effect size for Kappa's?

I am staring a project on comparing standard ways of creating a classifier with some heuristic methods. The heuristic methods should result in a faster training for the classifier but should result in ...
0
votes
0answers
12 views

Repeated CV vs. CV

Suppose a data set contains the numbers $1-100$ (i.e. $\{1, \dots, 100 \}$). In $10$-fold cross validation, the data set is divided into $10$ subsets with one used as the validation data set. For ...
0
votes
0answers
20 views

Constant RMSE margin between training and teseting set

I have a large number of independent datasets of varying size but same feature meaning. Features and outcome are both binary. I am trying to fit logistic regression to the data. I estimate ...
5
votes
2answers
108 views

AIC, BIC and GCV: what is best for making decision in penalized regression methods?

My general understanding is AIC deals with the trade-off between the goodness of fit of the model and the complexity of the model. $AIC =2k -2ln(L)$ $k$ = number of parameters in the model $L$ = ...
1
vote
0answers
11 views

CV for LASSO tuning parameter using LARS

If I use the LARS algorithm to fit the LASSO path, is it sufficient to cross-validate using the values of $\lambda$ at each step in LARS or is it better to use a finer grid of $\lambda$ values? I ...
0
votes
0answers
13 views

Getting the rule from cross validation

I've got a question. Let's say I have a medical data representing 2 classes of patients (healthy and unhealthy) and some number of predictors which characterize these patients. Choosing different ...
1
vote
0answers
32 views

What are appropriate validation methods for a Bayesian network model with low sample size?

I am currently using a Bayesian network model with 20 variables and 210 data points, with 15 locations measured at 14 different time points each. There are also some restrictions on what types of ...
1
vote
0answers
22 views

Is hold-out validation a better approximation of “getting new data” than k-fold CV?

I've been rethinking an answer I gave to a question a couple weeks ago Hold-out cross-validation produces a single test set that can be used repeatedly for demonstration. We all seem to agree that ...
0
votes
1answer
31 views

Interpretation of Output

Here is the R code that produced the output below: ...
1
vote
1answer
49 views

Is cross validation needed?

Suppose we have training data set and a test data set. The outcome variable is binary. Is it usually necessary to split the training data set so that there is a cross validation data set? Or can you ...
2
votes
2answers
32 views

The size of the sample for split validation

At this moment I have a dataset with 4000 samples (50% positive and 50% negative). Normally I would do cross validation for this approach, however besides normal data mining techniques I am also ...
0
votes
0answers
6 views

Liblinear logisitic regression with L2 regularization for classification

I am trying to use the liblinear library for logistic regression with L2 regularization. However, I am finding some issues with it. For eg when choosing the cost parameter, I chose the C parameter to ...
1
vote
0answers
29 views

Using Leave-One-Out Cross Validation with LARS

I have a kind of obscure question about using the Least Angle Regression (LARS) algorithm for variable selection. If I'm understanding it right, my professor formulates LARS as such: $$\mathbb{min}\ ...
7
votes
3answers
322 views

How is cross validation different from data snooping?

I just finished "An Introduction to Statistical Learning". I wondered whether using cross-validation to find the best tuning parameters for various machine learning techniques is different from data ...
0
votes
0answers
22 views

Are there any well-understood circumstances when we should expect cross-validation to be indifferent to $\alpha$ in elastic net tuning?

I am fitting a lot of elastic net models, simultaneously tuning $\lambda$ and $\alpha$. I am often coming to the following conclusions: Cross-validation error is much more sensitive to changes in ...
1
vote
0answers
7 views

Random seed choice changes qualitative results of elastic net parameter tuning

Below is a function I wrote to try and tune the $\lambda$ and $\alpha$ elastic net GLM implemented with cv.glmnet. I've noticed that the qualitative outcome (in ...
3
votes
1answer
66 views

Variance-covariance matrix for ridge regression with stochastic $\lambda$

In ridge regression with design matrix $X$, outcomes $y$, fixed regularization parameter $\lambda$, and errors $\epsilon\sim\mathcal{N}(0, \sigma^2I)$, the computations for the ridge regression ...
0
votes
0answers
33 views

Training and testing on Unbalanced Data Set

I used SMOTE algorithm in R for class balancing. My data size has 13000 rows, I had 7% minority class in my sample now I used SMOTE( Synthetic Minority Oversampling Technique) for class balancing such ...
0
votes
0answers
15 views

visualize the naive bayes with k-fold cv

I have done the classification using naive Bayes as a classifier, and applied 10-fold CV.I know that I can get the mean and variance of the result. However, how can I plot the classifier performance? ...
0
votes
0answers
45 views

using Root Mean Squared Error (RMSE) to compare models with different sample size

I'm using k-fold cross-validation to compare different models. I splitted my dataset in 6 chunks and used 4 random chunks as training set and the remaining 2 as a test set. Now I fitted n-different ...
0
votes
0answers
38 views

BRT analysis using count data

I have some problems with my BRT analysis. Introduction to the data: The dependent variable is count data of a specific palm species in SA, and the predictors consists of nine various kinds of ...
0
votes
1answer
29 views

Dividing up Training Data into Test Set

Suppose we have a training data set. We want to learn some hypothesis using some algorithm. Would we divide up the training set differently if we used, for example, logistic regression as opposed to ...
2
votes
0answers
27 views

Calculate prediction errors of binary model: What's a good way?

I am totally new to statistics, so this may be obvious, but I don't get it. Basicly, I fit a special kind of tree-model to a subset of data (one half), and now I want to cross-validate my model ...
1
vote
0answers
25 views

Cross-Validation vs. AICc for LASSO

I was working on a research project in which I try to estimate the the individual contribution of a group of regional political leaders to local economic growth. The major challenge is that there is ...
0
votes
0answers
33 views

R cv.glm returns NaN for stepwise-generated regression model

I'm trying to run K-fold cross-validation on a multiple regression model that was generated via the step function in R. However, the call to ...
5
votes
0answers
86 views

Minimum sample size for a dichotomous outcome

I have two questions. I am running an experiment where I am interested in determining the sample size required for a certain CI and error, where values range between $<1$ and $>-1$. However, I ...
0
votes
0answers
23 views

Question Regarding Nested-Cross Validation

I am slightly confused of the setup for tuning hyperparameters under gridsearch. Below is Python-style pseudo code for my understanding. ...
0
votes
0answers
12 views

Validating a multivariate categorical model

I assume that my population is a sample of an unknown multivariate categorical distribution $\mathbf{X} = (X_1, X_2, \ldots, X_k)$. From this population, a sample $\mathbf{X^*}$ is available, I assume ...
5
votes
3answers
213 views

K-fold or hold-out cross validation for ridge regression using R

I am working on cross-validation of prediction of my data with 200 subjects and 1000 variables. I am interested ridge regression as number of variables (I want to use) is greater than number of ...
0
votes
0answers
13 views

Neural nets method? Am i doing something wrong?

To set the scene - i am using neural nets in SAS program i have ~ 1000 points variables have been selected prior to this on a combination of human knowledge and variable recombination (PCA etc) I am ...
2
votes
2answers
81 views

Model Tuning and Model Evaluation in Machine Learning

Despite my readings (on stack 1, 2, or in literature (Cawley, 2010; Japkowicz, 2011)), I don't find a clear procedure for tuning and evaluating a model in a classification task. I want to perform a ...
9
votes
4answers
460 views

Hold-out Validation vs K-Fold Validation?

To me, it seems that Hold-out validation is useless. That is, splitting the original dataset into two-parts (training and testing) and using the testing score as a generalization measure, is somewhat ...
1
vote
0answers
24 views

Unstable models, repeated crossvalidation, feature selection

I'm still trying to classify few (about 200) samples in a high dimensional feature space (dim=19) into 3 (very unbalanced) classes. I use an implementation of Least Squares SVM with one vs one coding ...
0
votes
2answers
31 views

cross validation in ridge regression for classification. regularization issue

I perform ridge regression for classification. To find regularization parameter I do K-fold cross-validation with classification accuracy as a measure. This gives me some $\lambda$, which I then use ...
0
votes
0answers
13 views

Cross validation to test the performance of two spatial interpolation methods

I have 14 weather stations' temperature data for the period between 2010–2013. I need to evaluate the performance of two spatial interpolation methods. I suggest to select 10 days from this period of ...
2
votes
0answers
32 views

Cross validating quantile regression

I applied quantile regression on some data and did it for tau = 0.25, 0.5, 0.75. After i got the estimates of each model, i did some cross validation on my hold out data. When i used the estimates for ...
0
votes
0answers
21 views

Bias/Variance Trade-off in Cross-Validation

As I understand it, in K-fold cross validation, the bias gets smaller as K gets larger but the variance increases too. I'm having problems in intuitively understanding this concept from the variance ...
1
vote
1answer
43 views

Overfitting in Genetic Programming

I've recently started experimenting with Genetic Programming as an optimization tool. I'm still a little confused as to how to reduce overfitting in this framework. A couple of techniques I've read ...