0
votes
1answer
37 views

High Standard Deviation for Leave one out cross-validation?

I am using the leave one out cross-validation technique to evaluate my model. If the prediction on the test sample is right the output is 1 otherwise 0. So I have a array of N samples with 0's and 1's ...
0
votes
0answers
26 views

Validation accuracy larger than training accuracy

I was performing an experiment but got a higher validation accuracy than training accuracy. I've got a 39 mice data and performed leave one out cross-validation. The validation accuracy was 100%. But ...
0
votes
0answers
25 views

Minimizing the Training data

I have a grey-box model of the form Y= a + b X1 + c X2. Where a, b and c are the coefficients based on regression. The regression variables X1 and X2 are determined based on ...
0
votes
1answer
35 views

How do you measure the accuracy of an inference hypothesis/procedure?

Take inference to mean reasoning/predicting the value of a hidden/laten variable $Z$ given some evidence/data $X$. For example, maybe you are trying to find out if your patient has Cancer (Z = 1 if he ...
2
votes
2answers
105 views

About cross-validation for machine learning

Assume I have 1000 samples of data. I split the data randomly into training and test sets of size 800 and 200, respectively. Now, I train a classifier using the training set, and then evaluate the ...
3
votes
2answers
63 views

Assigning even partitions for Cross-Validation

This is a very basic question about cross-validation. Say that I have a sample size of 2901(or any difficult to divide number). How do I split this up into equal partitions (other than n=1)? And how ...
0
votes
0answers
54 views

10 fold cross validation model in weka

Trying to build a specific Neural Network arcitecture and testing it using 10 fold cross validation of a dataset. Now building the model is a tedious job and Weka expects me to make it 10 times for ...
1
vote
0answers
36 views

What are appropriate validation methods for a Bayesian network model with low sample size?

I am currently using a Bayesian network model with 20 variables and 210 data points, with 15 locations measured at 14 different time points each. There are also some restrictions on what types of ...
7
votes
3answers
378 views

How is cross validation different from data snooping?

I just finished "An Introduction to Statistical Learning". I wondered whether using cross-validation to find the best tuning parameters for various machine learning techniques is different from data ...
3
votes
1answer
77 views

Variance-covariance matrix for ridge regression with stochastic $\lambda$

In ridge regression with design matrix $X$, outcomes $y$, fixed regularization parameter $\lambda$, and errors $\epsilon\sim\mathcal{N}(0, \sigma^2I)$, the computations for the ridge regression ...
0
votes
0answers
42 views

Training and testing on Unbalanced Data Set

I used SMOTE algorithm in R for class balancing. My data size has 13000 rows, I had 7% minority class in my sample now I used SMOTE( Synthetic Minority Oversampling Technique) for class balancing such ...
2
votes
2answers
87 views

Model Tuning and Model Evaluation in Machine Learning

Despite my readings (on stack 1, 2, or in literature (Cawley, 2010; Japkowicz, 2011)), I don't find a clear procedure for tuning and evaluating a model in a classification task. I want to perform a ...
9
votes
4answers
504 views

Hold-out Validation vs K-Fold Validation?

To me, it seems that Hold-out validation is useless. That is, splitting the original dataset into two-parts (training and testing) and using the testing score as a generalization measure, is somewhat ...
2
votes
2answers
76 views

Use of nested cross-validation

Scikit Learn's page on Model Selection mentions the use of nested cross-validation: ...
0
votes
0answers
24 views

Cross validation and accuracy calculation in lib-linear

I have two questions related to cross validation in LIBLINEAR I have 1000 documents from which i take 300 documents for training and rest 700 for classification . I train 300 documents with ...
3
votes
2answers
105 views

How to find optimal values for the tuning parameters in boosting trees ?

I realise that there are 3 tuning parameters in the boosting trees model, i.e. the number of trees (number of iterations) shrinkage parameter number of splits (size of each constituent trees) My ...
0
votes
0answers
40 views

The correct way to do Cross-Validation

Consider the case that I need to do cross-validation for SVM to obtain a good estimate of the cost parameter $C$. I am not sure when should I divide the data into $K- $ folds. To perform the ...
0
votes
0answers
41 views

Correct methodology to repeat testing of classifier to get good estimate of performance

I'm having trouble with a basic machine learning methodology question. I understand the concept of not using the same data to both train and evaluate a classifier, and furthermore when there are ...
0
votes
1answer
36 views

How is the training set constructed for multi-class SVMs?

Support vector machines do binary classification. If there is more than two classes, it is possible to train several classifiers instead of one. Two common approaches are training one vs. one (each ...
2
votes
1answer
74 views

Real World Challenge: Large difference between training and testing set accuracy

I have a classification dataset of ~100,000 rows and ~200 features. Within the dataset my predictor variable (Y) is an integer value between 0-55, therefore I am trying to predict 1 of 56 possible ...
1
vote
0answers
35 views

How to use cross-validation [closed]

I have a 24983 X 100 matrix. The cell (i, j) in the matrix, indicates the rating for joke number j by user number i. I need to recommend to user i the joke would make him laugh the most. I need to ...
0
votes
1answer
145 views

Train / Validate / Test sets in Caret

I want to use caret to compare two different classification algorithms. For example SVM and Elastic net. I want to put aside some samples for test set and then use the rest of the samples for ...
0
votes
0answers
9 views

Which is the current testing methodology for pool-based active learning?

I have seen many papers using 10-fold CV ("10-pool" CV), but I think the accuracy obtained this way can be, sometimes, optimistically incorrect since at any given time step t the sum of distinct ...
0
votes
1answer
36 views

Does the standard deviation of the folds of LOO cross-validation have any practical meaning in comparison/evaluation of classifiers?

There are dozens of questions regarding LOO and variance. Most of the answers are purely theoretical or too general. I have also read many papers like this paper. Specifically: I have two not too ...
1
vote
2answers
75 views

What is v-fold cross validation?

What is v-fold cross validation in relation to k-fold cross validation? Also is there a more common way in which v-fold cross validation is referenced? I'm struggling to find resources on this ...
0
votes
0answers
41 views

Cross validation MATLAB code for multi-output regression

I am using multi-output support vector regression (MSVR) which predicts multiple outputs at a time. I want to know how can I initially select parameters for my model, and then how to do cross ...
2
votes
1answer
166 views

Final Model Prediction using K-Fold Cross-Validation and Machine Learning Methods

Similar threads: Feature selection for "final" model when performing cross-validation in machine learning Choosing a predictive model after k-fold cross-validation My question is quite ...
9
votes
1answer
408 views

How to split the dataset for cross validation, learning curve, and final evaluation?

What is an appropriate strategy for splitting the dataset? I ask for feedback on the following approach (not on the individual parameters like test_size or ...
1
vote
3answers
156 views

Leave One Out Cross Validation

I tried to implement the Leave One Out Cross Validation (LOOCV) method to get me a best combination of 4 data points to train my model which is of the form: Y= a + b X1 + c X2. Where a, b and c are ...
1
vote
0answers
62 views

Compare averaged GLM with boosted regression trees using cross validation : d2 and RMSE calculation

I want to compare BRT and averaged glm models on test sets by calculating the explained deviance and RMSE. How can I calculate d2 and RMSE from predictions? I use the following functions: gbm1 ...
2
votes
1answer
151 views

How to implement a hold-out validation in R

Let's say I'm using the Sonar data and I'd like to make a hold-out validation in R. I partitioned the data using the createFolds ...
0
votes
1answer
123 views

How can I perform 10-fold cross validation by manually constructing datasets?

I am working in text classification in RapidMiner where, because of the nature of my problem, I cannot use the built-in k-fold cross validation strategy, so I decided to create 10 copies of my dataset ...
0
votes
1answer
104 views

Grid Search for hyperparameter and feature selection

So I need to select my hyperparameters and also my features. A full grid search of the space of hyperparameters and features is too computationally intensive, so what I am doing instead is for each ...
1
vote
0answers
40 views

How to determine appropriate number of features and also which features to select?

So I have a dataset which I am using K fold cross validation on to select the number of features and which features should be selected. As I understand it, I would set the number of features to be ...
1
vote
0answers
36 views

What is the meaning of the term “enrichment” when performing cross-validation?

Trying to understand a discussion of a 5-fold cross-validation process to validate a predictive model and its results, there is a particular phrase which has me stumped, i.e.: The predictions of ...
1
vote
1answer
83 views

How is AUC of decision tree calculated?

I have a dataset which only has one continuous variable, and I try to use decision tree algorithm to build a model which classify the +ve and -ve label from the dataset. I run 10-fold ...
1
vote
0answers
41 views

Sizing of training and validation sets in machine learning: Is there a proven optimum, or merely heuristics?

When I watch presentations where machine learning algorithms were used, the amount of data put in the training and validation sets seems to be somewhat arbitrary. Sometimes it's 80-20, sometimes it's ...
1
vote
2answers
90 views

How to report a SVM model to a 3rd party after cross-validation?

I have a binary classification problem. I trained my dataset using a Support Vector Machine (SVM). Now I want to report the model I trained to a 3rd party so that they can use. For the primal probem ...
3
votes
2answers
266 views

k folds cross validation on a multi-class dataset

Cross validation is one of the most important tools because it gives us an honest assessment of the true accuracy of our system. In other words, the cross-validation process provides a much more ...
4
votes
3answers
233 views

Can you compare different clustering methods on a dataset with no ground truth by cross-validation?

Currently, I am trying to analyze a text document dataset that has no ground truth. I was told that you can use k-fold cross validation to compare different clustering methods. However, the examples I ...
4
votes
3answers
318 views

Is a lower training accuracy possible in overfitting (one class SVM)

I am using the heart_scale data from LibSVM. The original data includes 13 features, but I only used 2 of them in order to plot the distributions in a figure. Instead of training the binary ...
0
votes
1answer
44 views

How to compare features and classifiers which achieve perfect accuracy?

So I'm looking to compare different combinations of features and classifiers. But I'm getting a lot of combinations that achieve 100% cross validation accuracy. I'm trying to figure out how I would ...
3
votes
3answers
41 views

How to compare features and classifiers which achieve perfect accuracy?

So I'm looking to compare different combinations of features and classifiers. But I'm getting a lot of combinations that achieve 100% cross validation accuracy. I'm trying to figure out how I would ...
0
votes
0answers
32 views

Problem with classifier prediction results

I built a classifier with 13 features ( no binary ones ) and normalized individually for each sample using scikit tool ( Normalizer().transform). When I make predictions it predicts all training sets ...
5
votes
2answers
267 views

Why is k-fold cross validation a better idea than k-times resampling true validation?

I'm currently working through a machine learning textbook and just read a bit about k-fold cross validation, and I am wondering the following. I want to estimate a parameter, e.g. a penalty parameter ...
0
votes
0answers
102 views

Understanding the RandomForest with 10x10 cross validation for classification

I'm trying to understand the built of a random forest with the $10\times10$ cross validation for a binary classification problem. Therefore I have 4 basic questions: Notation: $N=500$ trees $i=$ ...
4
votes
1answer
702 views

How to choose train/test sample ratio, for machine learning?

I am building a real time machine learning module, which is not based on a huge** sample size, with hyper parameter grid search and cross validation process. I am looking for any insight/advice, as ...
10
votes
3answers
483 views

Cross-validation including training, validation, and testing. Why do we need three subsets?

I have a question regarding the Cross-validation process. I am in the middle of a course of the Machine Learning on the Cursera. One of the topic is about the Cross-validation. I found it slightly ...
0
votes
1answer
154 views

Feature selection and cross validation

I'm working on a project and I would like to know if the following strategy is good/correct. Sorry if this is a basic/stupid idea (I'm new to this). The input is a dataset with 2.500 features and ...
0
votes
0answers
54 views

Model selection for unbalanced data

How to do model selection for unbalanced data? how many data points from the whole data set should be selected for model selection? how many for training and testing?