Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
19 views

Random Forest - understanding k fold cross validation

I am trying to improve my data science knowledge by solving problems available on the internet. I am currently using the R package randomForest to classify the ...
0
votes
0answers
14 views

Choosing fold size for highly Imbalanced dataset + nested CV + svm

I am trying to classify a dataset with ~1000 points. 90/10 is the class ratio - super imbalanced. Here are the following steps I did: Use 20 relevant features from previous knowledge Remove highly ...
0
votes
0answers
23 views

R - Random Forest - Need help understanding the rfcv function

My name is Abhi. I am trying to teach myself data science by solving some of the problems available on the internet. My current data set has about 900 reccords & 10 features. I am trying to use ...
0
votes
1answer
50 views

High Standard Deviation for Leave one out cross-validation?

I am using the leave one out cross-validation technique to evaluate my model. If the prediction on the test sample is right the output is 1 otherwise 0. So I have a array of N samples with 0's and 1's ...
1
vote
0answers
15 views

Cross validation with nonparametric smoothing regressions

When I use regression models I like to explore functional relationships using nonparametric smoothing regression (e.g. generalized additive models, lowess, running line smoothers, etc.) before ...
0
votes
0answers
16 views

Triple nested cross validation

I have read several very informative posts including the link about the nested/double cross validation, which can determine (sub)optimal hyperparameter values as well as make an unbiased estimate of ...
0
votes
1answer
51 views

Linear model- Understanding performances on training and test sets

I have a small normalized data set, 30 observations and 18 Predictors. All are continuous and some variable are related. I ran linear regression on it using Weka. The model automatically dropped some ...
0
votes
0answers
27 views

Validation accuracy larger than training accuracy

I was performing an experiment but got a higher validation accuracy than training accuracy. I've got a 39 mice data and performed leave one out cross-validation. The validation accuracy was 100%. But ...
2
votes
1answer
42 views

Cross-validation with dummy variables?

Does it make sense to use cross-validation with factor variables that have 3+ levels? When using bestglm, I get an error saying that it doesn't work with categorical variables. In the documentation ...
0
votes
0answers
15 views

How to forecast with quantile regressoin

If you have three quantile regression models with taus of 0.25, 0.5 and 0.75 and their coefficients how do you use these models to forecast a set of data not used to calculate the coefficients. In ...
0
votes
0answers
25 views

Parameter optimization of SVM

Currently I am using SVM to perform some classification task. I use libSVM with Matlab interface. From the practical guide of SVM (Link), we know that there are two parameters need to be tuned, namely ...
0
votes
0answers
9 views

Weka: cross validation using blocks of related instances (leave one patient out)

I have a dataset that comprises several instances for different patients, with multiple instances per patient. I need to perform some classification tasks and I was using cross-validation, but this ...
1
vote
1answer
19 views

Does it make sense to calculate Q2 and R2 values on PLS-DA models?

Since PLS-DA is a computational technique which deals with outcomes expressed as a categorical variable (e.g. "Yellow","Brown","Black","Green") I cannot understand how it is possible to calculate Q2 ...
2
votes
2answers
38 views

Does it make sense to do CV-error-weighted model averaging?

We often average models together to create an aggregate prediction model. Some recent research suggests that simple model averages perform as well or better than model averages weighted by functions ...
0
votes
1answer
13 views

how to impute missing values on numpy array created by train_test_split from pandas.DataFrame?

I'm working on the dataset with lots of NA values with sklearn and pandas.DataFrame. I implemented different imputation strategies for different columns of the dataFrame based column names. For ...
2
votes
2answers
44 views

Is cross validation for validating a model or for selecting best model in different kinds of models?

I am confused about the concept of cross validation and its usage. As I read about cross validation before, it is a way of validating a model. I did cross validation in my project (developing ...
0
votes
0answers
18 views

Can I use cross-validation to select optimal parameters SEPARATELY?

I'm wondering if there is any math/stat theory out there to support or deny this idea: I am using cross-validation and building models over a vector of parameter values to then choose the optimal ...
0
votes
0answers
24 views

Leave-one-out cross validation in selecting predictor

I am a newbie here. There are 155 total samples. Five different predictors Xi (i=1,2...5) are used to predict Y, like X1 X2 X3 X4 X5 Y .... The objective is to find the best predictor Xi to ...
0
votes
1answer
22 views

Caret: customizing feature selection, nested inside cross validation

Using caret, I want to train a SVM classifier and estimate its performance using repeated cross validation. My dataset has a very large number of predictors (300K) and I want to reduce this number ...
0
votes
0answers
25 views

Minimizing the Training data

I have a grey-box model of the form Y= a + b X1 + c X2. Where a, b and c are the coefficients based on regression. The regression variables X1 and X2 are determined based on ...
1
vote
1answer
265 views

Which is kernel similar gaussian kernel?

I must find a kernel that statisfies as follows: In the my reference paper, the author suggest gaussian kernel that is The purpose of that kernel is that it will take a weight for each points ...
0
votes
0answers
11 views

Parameter estimation of gaussian function kernel using cross-validation

I need to estimate (using cross-validation), the parameters $\sigma$ and $\lambda$ of the Gaussian kernel: $K_G(x,y) = \sigma^2 \exp{(-\frac{1}{2\lambda^2}\sum_{i,j}(x_{ij}-y_{ij})^2})$ where $x$ ...
0
votes
2answers
27 views

Cross Vaidation in Factor Analysis

I am trying to conduct an EFA with a sample size of 150 respondents. I would also like to use cross-validation but my professor says that the sample is not big enough for that. Is that true?
2
votes
1answer
69 views

Why should all Cross-Validation results be higher than the result on the test dataset?

Sorry, I'm not an expert and my question could be fundamentally wrong. I've read this interesting question because I also was wondering whether to train the model again after cross-validation. Now, ...
0
votes
1answer
35 views

How do you measure the accuracy of an inference hypothesis/procedure?

Take inference to mean reasoning/predicting the value of a hidden/laten variable $Z$ given some evidence/data $X$. For example, maybe you are trying to find out if your patient has Cancer (Z = 1 if he ...
0
votes
1answer
26 views

Feature Selection using (low) MCC

I have approximately 1200 input parameters that I am trying to whittle down with the following rough process: 1) Fit rbf SVM with n = 1200 parameters and calculate Matthews Correlation ...
0
votes
1answer
25 views

Predictive modeling techniques for in-sample rather than out-of-sample prediction?

Is it appropriate to apply predictive modeling variable selection and shrinkage techniques (for example, ridge regression or lasso) for in-sample prediction rather than out-of-sample prediction? ...
3
votes
2answers
108 views

About cross-validation for machine learning

Assume I have 1000 samples of data. I split the data randomly into training and test sets of size 800 and 200, respectively. Now, I train a classifier using the training set, and then evaluate the ...
3
votes
2answers
64 views

Assigning even partitions for Cross-Validation

This is a very basic question about cross-validation. Say that I have a sample size of 2901(or any difficult to divide number). How do I split this up into equal partitions (other than n=1)? And how ...
2
votes
0answers
20 views

Use of Deviance Residuals for Leave-One-Out Cross Validation

I am a newbie to stats and having some difficulties understanding how to use deviance residuals for leave-one-out cross validation for a logistic regression model. The problem that I am trying to ...
0
votes
1answer
29 views

Leave-one-subject-out cross validation in Caret

Hi Dear Colleagues, I wonder how to correctly setup a leave-one-subject-out cross validation (LOSO) for train() function in caret. Here is my example code: ...
0
votes
0answers
55 views

10 fold cross validation model in weka

Trying to build a specific Neural Network arcitecture and testing it using 10 fold cross validation of a dataset. Now building the model is a tedious job and Weka expects me to make it 10 times for ...
0
votes
0answers
16 views

Binary classification with too few positive samples

I met a problem of doing a binary classification with quite few positive samples. For example: Binary classification with either labelled 1 samples (positive) or labelled 0 samples (negative ...
0
votes
0answers
20 views

Repeated CV vs. CV

Suppose a data set contains the numbers $1-100$ (i.e. $\{1, \dots, 100 \}$). In $10$-fold cross validation, the data set is divided into $10$ subsets with one used as the validation data set. For ...
0
votes
0answers
22 views

Constant RMSE margin between training and teseting set

I have a large number of independent datasets of varying size but same feature meaning. Features and outcome are both binary. I am trying to fit logistic regression to the data. I estimate ...
5
votes
2answers
136 views

AIC, BIC and GCV: what is best for making decision in penalized regression methods?

My general understanding is AIC deals with the trade-off between the goodness of fit of the model and the complexity of the model. $AIC =2k -2ln(L)$ $k$ = number of parameters in the model $L$ = ...
1
vote
0answers
13 views

CV for LASSO tuning parameter using LARS

If I use the LARS algorithm to fit the LASSO path, is it sufficient to cross-validate using the values of $\lambda$ at each step in LARS or is it better to use a finer grid of $\lambda$ values? I ...
0
votes
0answers
15 views

Getting the rule from cross validation

I've got a question. Let's say I have a medical data representing 2 classes of patients (healthy and unhealthy) and some number of predictors which characterize these patients. Choosing different ...
1
vote
0answers
36 views

What are appropriate validation methods for a Bayesian network model with low sample size?

I am currently using a Bayesian network model with 20 variables and 210 data points, with 15 locations measured at 14 different time points each. There are also some restrictions on what types of ...
1
vote
0answers
30 views

Is hold-out validation a better approximation of “getting new data” than k-fold CV?

I've been rethinking an answer I gave to a question a couple weeks ago Hold-out cross-validation produces a single test set that can be used repeatedly for demonstration. We all seem to agree that ...
0
votes
1answer
35 views

Interpretation of Output

Here is the R code that produced the output below: ...
2
votes
2answers
73 views

Is cross validation needed?

Suppose we have training data set and a test data set. The outcome variable is binary. Is it usually necessary to split the training data set so that there is a cross validation data set? Or can you ...
2
votes
2answers
35 views

The size of the sample for split validation

At this moment I have a dataset with 4000 samples (50% positive and 50% negative). Normally I would do cross validation for this approach, however besides normal data mining techniques I am also ...
0
votes
0answers
10 views

Liblinear logisitic regression with L2 regularization for classification

I am trying to use the liblinear library for logistic regression with L2 regularization. However, I am finding some issues with it. For eg when choosing the cost parameter, I chose the C parameter to ...
1
vote
0answers
30 views

Using Leave-One-Out Cross Validation with LARS

I have a kind of obscure question about using the Least Angle Regression (LARS) algorithm for variable selection. If I'm understanding it right, my professor formulates LARS as such: $$\mathbb{min}\ ...
7
votes
3answers
383 views

How is cross validation different from data snooping?

I just finished "An Introduction to Statistical Learning". I wondered whether using cross-validation to find the best tuning parameters for various machine learning techniques is different from data ...
0
votes
0answers
26 views

Are there any well-understood circumstances when we should expect cross-validation to be indifferent to $\alpha$ in elastic net tuning?

I am fitting a lot of elastic net models, simultaneously tuning $\lambda$ and $\alpha$. I am often coming to the following conclusions: Cross-validation error is much more sensitive to changes in ...
1
vote
0answers
10 views

Random seed choice changes qualitative results of elastic net parameter tuning

Below is a function I wrote to try and tune the $\lambda$ and $\alpha$ elastic net GLM implemented with cv.glmnet. I've noticed that the qualitative outcome (in ...
3
votes
1answer
77 views

Variance-covariance matrix for ridge regression with stochastic $\lambda$

In ridge regression with design matrix $X$, outcomes $y$, fixed regularization parameter $\lambda$, and errors $\epsilon\sim\mathcal{N}(0, \sigma^2I)$, the computations for the ridge regression ...
0
votes
0answers
43 views

Training and testing on Unbalanced Data Set

I used SMOTE algorithm in R for class balancing. My data size has 13000 rows, I had 7% minority class in my sample now I used SMOTE( Synthetic Minority Oversampling Technique) for class balancing such ...