Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
3answers
17 views

Finding the optimal value of k in the k-nearest-neighbor classifier: is this cross-validation?

I have collected 1000 data points with each data point belonging to eight categories. I would like to be able to correctly estimate the categories of any new data by using the k-nearest-neighbor ...
1
vote
0answers
8 views

Hierarchical Cluster Analysis validation

I have never used Hierarchical Cluster Analysis for inferential statistics before, but the dendrogram it produces provides a nice way to visualise my data. I applied the HCA to my variables with the ...
0
votes
0answers
11 views

How to perform multiclass SVM classification using k-fold cross-validation and SMO with some kernel method in MATLAB?

I have data matrix X.csv file of size nxd, where n are the observations, and d variables. There are c_1,..., c_m classes. Let, Y be the matrix containing the class labels. There is no header row in ...
0
votes
0answers
21 views

Using confidence intervals with Simple Linear Regression

So simple linear regression is performed on 3000 data points, and 1000 data points are withheld. How can we use confidence intervals, along with the withheld data points, to assess the predictive ...
0
votes
0answers
14 views

Cross validated loglikelihood?

This is probably a silly question: I was playing around with penalized package and cvl outputs a cross validated loglikelihood and another measure just called loglikelihood which is suppose to be ...
0
votes
0answers
30 views

Sum of two different probability functions [on hold]

I'm have some problems estimating the marginal distribution from data. I found three different distributions that explain the data quite well: two log-normal (black) and one normal distribution (red). ...
0
votes
0answers
9 views

f1 score of all classes from scikits cross_val_score [on hold]

I'm using cross_val_score from scikit-learn (package sklearn.cross_validation) to evaluate my classifiers. If I use ...
0
votes
0answers
30 views

Scaling the data in a decision tree changed my results?

I know that a decision tree doesn't get affected by scaling the data but when I scale the data within my decision tree it gives me a bad performance (bad recall, precision and accuracy) But when I ...
0
votes
1answer
20 views

Confidence and Prediction Intervals and Cross-Validation

So suppose, I have 5000 points of data. I hold out 1000 for validation testing, and conduct simple linear regression on the first 4000 points. How can I determine if the linear model: y = a + bx + ...
2
votes
0answers
13 views

leave-h-out cross validation

I'm doing multistep forecasts of univariate time series and a wide range of exogenous leading indicator variables are available. Therefore I'm looking for ways to optimally select and/or combine ...
3
votes
2answers
89 views

k fold cross validation: nominal predictor level appears in the test data but not the training data

I wrote my own cross validation function for model output in R (lm, glm and so on; named ...
0
votes
1answer
22 views

parameter tuning using nested cross validation

Parameter tuning in SVM has been performed using a nested cross-validation(CV) approach with 45 folds(outer loop) and 13 folds(inner loop). In this process, the outer loop will have 45 prediction ...
0
votes
0answers
6 views

Cross validation of dataset separated on files [migrated]

The dataset that I have is separated on different files grouped on samples that know each other, i.e., they were created on similar conditions on a similar time. The balance of the train-test dataset ...
0
votes
1answer
33 views

10-fold cross validation on small number of examples

I have a set of 100 examples evaluated using 10-fold cross validation, providing 94% classification accuracy on the test folds. However, when I test the model on a different test set, it provides 0% ...
0
votes
1answer
34 views

L2-regularized MLR using caret and how to make sure I am using the best tuned model

I am trying to do L2-regularized MLR on a data set using caret. Following is what I have done so far to achieve this: ...
14
votes
5answers
362 views

On the importance of the i.i.d. assumption in statistical learning

In statistical learning, implicitly or explicitly, one always assumes that the training set $\mathcal{D} = \{ \bf {X}, \bf{y} \}$ is composed of $N$ input/response tuples $({\bf{X}}_i,y_i)$ that are ...
0
votes
1answer
74 views

Cross-validation of multiple subjects with multiple instances

I have a training set of 50 subjects with about 550-600 measurements each. One measurement consists of 24 features and one class label (1 or 0). So my data looks like this (simplified): ...
0
votes
0answers
25 views

How to evaluate model improvement using cross validation?

I have two logistic regression models A and B. A is nested under B, i.e., A's features are the subset of B's features. To evaluate both models, I use 10-fold cross validation: (1) train A and B on ...
0
votes
0answers
35 views

Cross-validation in combination with Multiple Imputation

I am working on a project where I want to cross-validate a Machine Learning algorithm (not logistic regression) on multiply imputed data. My question is, how can I use the training data to multiply ...
0
votes
1answer
23 views

Cross Validation with Replicates

I have a question about performing either 10-fold or leave one out cross validations with biological replicates. In total I have 50 samples, each of which has four biological replicates. I am ...
0
votes
0answers
25 views

How do I optimize a bioinformatics pipeline for novel data sets?

I'm putting the finishing touches on a bioinformatics pipeline for omics data. There are many sequential interlocking parts (e.g. model fitting, regression, classification, clustering, etc). The final ...
0
votes
1answer
21 views

K-fold CV based model selection with a constraint on the number of features?

I am currently working on project where I need to train a logistic regression classifier with a combined $l_1$/$l_2$-penalty that satisfies a hard on the number of features. Specifically, my dataset ...
1
vote
0answers
15 views

Replicates handling in cross-validation

Currently, I try to analyze the gene expression data from approximately 1000 different samples, so I obtain a matrix of dimension 20000 (no. of genes) x 1000 (no. of samples), where each entry ...
0
votes
0answers
32 views

Handling imbalanced datasets and misclassification costs in SVMs?

I have a dataset with 50 times more negative examples than positive ones. Currently, I am using an oversampling technique to address the imbalance problem. During the model selection stage (i.e. ...
0
votes
0answers
18 views

Can we use cross validation and bootstrapping together?

I would like to estimate the model parameters from n data samples in a training data set. I want to know if I can use bootstrap and cross validation jointly. For instance, I have n data samples. ...
1
vote
2answers
21 views

Equal sized sets in cross-validation

What is the advantage you get out of choosing equal sized training and test sets in cross-validation? Why don't you split your set in two sets of different size in 2-fold cross-validation for example? ...
0
votes
2answers
30 views

Estimating classifier performance using cross validation, average accuracy and standard deviation and

I want to estimate a classifier accuracy on benchmark data. Data is not split into training and testing so I use 5-fold cross validation, using 80% of data as training and testing on 20%. Each test is ...
0
votes
1answer
37 views

Training and test sets in random forest regression

Why do I keep reading about specifying training and test sets with random forest regression? As probably obvious from this question I am new to this method, but what I thought one of the cool things ...
3
votes
1answer
53 views

Cross validation after LASSO in complex survey data

I am trying to do model selection on some candidate predictors using LASSO with a continuous outcome. The goal is to select the optimal model with the best prediction performance, which usually can be ...
3
votes
2answers
73 views

Do I do threshold selection for my logit model on the testing or training subset?

I have data with a binary outcome and I am doing logit model selection using AIC and BIC. I have already withheld 30% of the data as a holdout sample (testing subset) and used the remainder (training ...
0
votes
0answers
20 views

Computing leave-one-out score of the linear regression for a large-scale regression

I heard that, for a linear regression, a leave-one-out cross validation score can be written in an explicit formula (using a matrix multiplication). (I browsed, e.g., ...
0
votes
0answers
74 views

cross validating time series model

I'm evaluating a model predicting clickrate. I have series of clicks and impressions. At the moment I'm evaluating a particular feature X. It's a categorical ...
0
votes
1answer
22 views

Clarification about not performing feature engineering before selecting CV folds

I'm new to stats, and I'm researching the pitfalls to avoid when performing Cross Validation (with classification). Suppose I'm training a classifier, and I have a dataset of 1000 samples, with 1 ...
1
vote
2answers
26 views

Validation set in presence of cross-validation

I am new to machine learning and want to ask regarding a confusion I have. I have a data set which is labeled and I want to do supervised learning. My question is related to cross-validation and ...
1
vote
2answers
43 views

Calculating misclassification rate for k-fold cross validation (logistic regression)

I am trying to manually write code to perform a k-fold cross validation for a logistic regression model for the first time. Unfortunately I am getting stuck trying to implement the following formula ...
1
vote
1answer
29 views

Parameter selection and k-fold cross validation

I have one dataset, and need to do cross-validation, for example, a 10-fold cross-validation, on the entire dataset. I would like to use radial basis function (RBF) kernel with parameter selection ...
0
votes
2answers
21 views

Cross-validation equal error for multiple parameter sets

When performing k-fold cross-validation on a training set, what is the best way to handle a minimum cv error that occurs for multiple sets of different parameters. Should all optimal parameters be ...
0
votes
2answers
50 views

Training set, test set and validation set

In order to do a data mining work I have to find the best classifiers for my data. What I wonder is if I have to divide my data set into a training test and a test set ? I have to choose between 5 ...
1
vote
1answer
38 views

Learning curves - Why does the training accuracy start so high, then suddenly drop?

I implemented a model in which I use Logistic Regression as classifier and I wanted to plot the learning curves for both training and test sets to decide what to do next in order to improve my model. ...
2
votes
2answers
56 views

Lasso and Ridge tuning parameter scope

In ridge and lasso linear regression, an important step is to choose the tuning parameter lambda, often I use grid search on log scale from -6->4, it works well on ridge, but on lasso, should I take ...
0
votes
0answers
12 views

using cross validation to produce and test a model [duplicate]

I'm a bit confused about cross validation purpose. lets say I have a linear model. one option is to get sample an split to train and test data sets. train on on train set, evaluate error on test set. ...
0
votes
1answer
42 views

What should be validation strategy?

I am building CTR(https://en.wikipedia.org/wiki/Click-through_rate) Click prediction model with different (61) variables.Dependent variable is weather 0/1( click).I have build logistic regression ...
0
votes
0answers
14 views

Is it common to perform a calcuation in several passes (similar to a k-fold cross validation)

i'm writting a paper where i want to plot the results of a similarity calcuation of a larger text corpus. (The result is a square matrix containing all the similarities between the documents) For each ...
0
votes
0answers
8 views

How to assign defined training set, val set and test set for training a Neural net in NNtoolbox?

To find an optimal number of hidden neurons and layers in my code using feedforward net, I use cross validation technique and cvpartition function to split data. Now my aim is to use this split data ...
3
votes
2answers
62 views

Completely different results after each cross validation

I'm running some classification algorithms in MATLAB and validating them with a 10-fold cross validation. The problem is that every time I execute the cross validation, it gives a very different ...
4
votes
2answers
57 views

Meaning of cross validation

This is a very fundamental question but I want to make sure I get this right. K-fold cross validation will only help in predicting the accuracy and other metrics of the model but not really improve ...
1
vote
1answer
34 views

Selecting a loss-function for k-fold cross-validation over shrinkage parameter

I am doing a penalized regression with categorical (ordinal) outcomes. I would like to select the shrinkage parameter $\lambda$ on the basis of cross-validation (CV). In this case, I have 50k ...
0
votes
0answers
24 views

How Cross validation for GLASSO works?

I just found a written R function to do cross validation for glasso to choose best lambda. ...
0
votes
2answers
35 views

Overfitting over test set in terms of model selection?

I didn't find any similar question so I just post the question here. Suppose after training and validation, the model performs poorly on the test set. Then what we do is to consider another model (Is ...