0
votes
2answers
21 views

Varying LIBSVM predictions based on test series labels

So I have a pretty well testing SVC train series which puts me into the mid 80 percentile without outrageous C/g values. My current C value is 2.0 and gamma is 0.5. Good numbers across the range ...
1
vote
1answer
28 views

Working with few data examples

I have been asked often in some interview, that how we should proceed when we have less data examples(say 50 or 100). What considerations needs to be made while choosing any algorithm. few points ...
0
votes
0answers
9 views

cross validation for kmodes in r

I am using k-modes (link) from the KlaR library (link) to cluster text data. I am not sure how to determine predictive error and thus perform cross-validation. Here is the "toy" sample, lets use ...
0
votes
1answer
38 views

Selecting most realistic C and g params after gridsearch

I just ran an extended SVC gridsearch in libsvm on about 9000 multi-dimensional vectors representing a time series. Here are the highest scoring results: ...
1
vote
2answers
54 views

How to choose the training, cross-validation, and test set sizes for small sample-size data?

Assume I have a small sample size, e.g. N=100, and two classes. How should I choose the training, cross-validation, and test set sizes for machine learning? I would intuitively pick Training set ...
0
votes
0answers
21 views

Choosing fold size for highly Imbalanced dataset + nested CV + svm

I am trying to classify a dataset with ~1000 points. 90/10 is the class ratio - super imbalanced. Here are the following steps I did: Use 20 relevant features from previous knowledge Remove highly ...
0
votes
1answer
75 views

High Standard Deviation for Leave one out cross-validation?

I am using the leave one out cross-validation technique to evaluate my model. If the prediction on the test sample is right the output is 1 otherwise 0. So I have a array of N samples with 0's and 1's ...
0
votes
0answers
29 views

Validation accuracy larger than training accuracy

I was performing an experiment but got a higher validation accuracy than training accuracy. I've got a 39 mice data and performed leave one out cross-validation. The validation accuracy was 100%. But ...
0
votes
0answers
25 views

Minimizing the Training data

I have a grey-box model of the form Y= a + b X1 + c X2. Where a, b and c are the coefficients based on regression. The regression variables X1 and X2 are determined based on ...
0
votes
1answer
46 views

How do you measure the accuracy of an inference hypothesis/procedure?

Take inference to mean reasoning/predicting the value of a hidden/laten variable $Z$ given some evidence/data $X$. For example, maybe you are trying to find out if your patient has Cancer (Z = 1 if he ...
3
votes
2answers
172 views

About cross-validation for machine learning

Assume I have 1000 samples of data. I split the data randomly into training and test sets of size 800 and 200, respectively. Now, I train a classifier using the training set, and then evaluate the ...
3
votes
2answers
69 views

Assigning even partitions for Cross-Validation

This is a very basic question about cross-validation. Say that I have a sample size of 2901(or any difficult to divide number). How do I split this up into equal partitions (other than n=1)? And how ...
0
votes
0answers
123 views

10 fold cross validation model in weka

Trying to build a specific Neural Network arcitecture and testing it using 10 fold cross validation of a dataset. Now building the model is a tedious job and Weka expects me to make it 10 times for ...
1
vote
0answers
43 views

What are appropriate validation methods for a Bayesian network model with low sample size?

I am currently using a Bayesian network model with 20 variables and 210 data points, with 15 locations measured at 14 different time points each. There are also some restrictions on what types of ...
7
votes
3answers
392 views

How is cross validation different from data snooping?

I just finished "An Introduction to Statistical Learning". I wondered whether using cross-validation to find the best tuning parameters for various machine learning techniques is different from data ...
3
votes
1answer
82 views

Variance-covariance matrix for ridge regression with stochastic $\lambda$

In ridge regression with design matrix $X$, outcomes $y$, fixed regularization parameter $\lambda$, and errors $\epsilon\sim\mathcal{N}(0, \sigma^2I)$, the computations for the ridge regression ...
0
votes
0answers
63 views

Training and testing on Unbalanced Data Set

I used SMOTE algorithm in R for class balancing. My data size has 13000 rows, I had 7% minority class in my sample now I used SMOTE( Synthetic Minority Oversampling Technique) for class balancing such ...
2
votes
2answers
97 views

Model Tuning and Model Evaluation in Machine Learning

Despite my readings (on stack 1, 2, or in literature (Cawley, 2010; Japkowicz, 2011)), I don't find a clear procedure for tuning and evaluating a model in a classification task. I want to perform a ...
9
votes
4answers
607 views

Hold-out Validation vs K-Fold Validation?

To me, it seems that Hold-out validation is useless. That is, splitting the original dataset into two-parts (training and testing) and using the testing score as a generalization measure, is somewhat ...
3
votes
2answers
130 views

Use of nested cross-validation

Scikit Learn's page on Model Selection mentions the use of nested cross-validation: ...
0
votes
0answers
27 views

Cross validation and accuracy calculation in lib-linear

I have two questions related to cross validation in LIBLINEAR I have 1000 documents from which i take 300 documents for training and rest 700 for classification . I train 300 documents with ...
3
votes
2answers
148 views

How to find optimal values for the tuning parameters in boosting trees ?

I realise that there are 3 tuning parameters in the boosting trees model, i.e. the number of trees (number of iterations) shrinkage parameter number of splits (size of each constituent trees) My ...
0
votes
0answers
41 views

The correct way to do Cross-Validation

Consider the case that I need to do cross-validation for SVM to obtain a good estimate of the cost parameter $C$. I am not sure when should I divide the data into $K- $ folds. To perform the ...
0
votes
0answers
44 views

Correct methodology to repeat testing of classifier to get good estimate of performance

I'm having trouble with a basic machine learning methodology question. I understand the concept of not using the same data to both train and evaluate a classifier, and furthermore when there are ...
1
vote
1answer
42 views

How is the training set constructed for multi-class SVMs?

Support vector machines do binary classification. If there is more than two classes, it is possible to train several classifiers instead of one. Two common approaches are training one vs. one (each ...
2
votes
1answer
82 views

Real World Challenge: Large difference between training and testing set accuracy

I have a classification dataset of ~100,000 rows and ~200 features. Within the dataset my predictor variable (Y) is an integer value between 0-55, therefore I am trying to predict 1 of 56 possible ...
1
vote
0answers
37 views

How to use cross-validation [closed]

I have a 24983 X 100 matrix. The cell (i, j) in the matrix, indicates the rating for joke number j by user number i. I need to recommend to user i the joke would make him laugh the most. I need to ...
0
votes
1answer
210 views

Train / Validate / Test sets in Caret

I want to use caret to compare two different classification algorithms. For example SVM and Elastic net. I want to put aside some samples for test set and then use the rest of the samples for ...
0
votes
0answers
9 views

Which is the current testing methodology for pool-based active learning?

I have seen many papers using 10-fold CV ("10-pool" CV), but I think the accuracy obtained this way can be, sometimes, optimistically incorrect since at any given time step t the sum of distinct ...
0
votes
1answer
48 views

Does the standard deviation of the folds of LOO cross-validation have any practical meaning in comparison/evaluation of classifiers?

There are dozens of questions regarding LOO and variance. Most of the answers are purely theoretical or too general. I have also read many papers like this paper. Specifically: I have two not too ...
1
vote
2answers
75 views

What is v-fold cross validation?

What is v-fold cross validation in relation to k-fold cross validation? Also is there a more common way in which v-fold cross validation is referenced? I'm struggling to find resources on this ...
0
votes
0answers
51 views

Cross validation MATLAB code for multi-output regression

I am using multi-output support vector regression (MSVR) which predicts multiple outputs at a time. I want to know how can I initially select parameters for my model, and then how to do cross ...
2
votes
1answer
205 views

Final Model Prediction using K-Fold Cross-Validation and Machine Learning Methods

Similar threads: Feature selection for "final" model when performing cross-validation in machine learning Choosing a predictive model after k-fold cross-validation My question is quite ...
10
votes
1answer
600 views

How to split the dataset for cross validation, learning curve, and final evaluation?

What is an appropriate strategy for splitting the dataset? I ask for feedback on the following approach (not on the individual parameters like test_size or ...
1
vote
3answers
172 views

Leave One Out Cross Validation

I tried to implement the Leave One Out Cross Validation (LOOCV) method to get me a best combination of 4 data points to train my model which is of the form: Y= a + b X1 + c X2. Where a, b and c are ...
1
vote
0answers
77 views

Compare averaged GLM with boosted regression trees using cross validation : d2 and RMSE calculation

I want to compare BRT and averaged glm models on test sets by calculating the explained deviance and RMSE. How can I calculate d2 and RMSE from predictions? I use the following functions: gbm1 ...
2
votes
1answer
175 views

How to implement a hold-out validation in R

Let's say I'm using the Sonar data and I'd like to make a hold-out validation in R. I partitioned the data using the createFolds ...
0
votes
1answer
142 views

How can I perform 10-fold cross validation by manually constructing datasets?

I am working in text classification in RapidMiner where, because of the nature of my problem, I cannot use the built-in k-fold cross validation strategy, so I decided to create 10 copies of my dataset ...
0
votes
1answer
129 views

Grid Search for hyperparameter and feature selection

So I need to select my hyperparameters and also my features. A full grid search of the space of hyperparameters and features is too computationally intensive, so what I am doing instead is for each ...
1
vote
0answers
41 views

How to determine appropriate number of features and also which features to select?

So I have a dataset which I am using K fold cross validation on to select the number of features and which features should be selected. As I understand it, I would set the number of features to be ...
1
vote
0answers
40 views

What is the meaning of the term “enrichment” when performing cross-validation?

Trying to understand a discussion of a 5-fold cross-validation process to validate a predictive model and its results, there is a particular phrase which has me stumped, i.e.: The predictions of ...
1
vote
1answer
98 views

How is AUC of decision tree calculated?

I have a dataset which only has one continuous variable, and I try to use decision tree algorithm to build a model which classify the +ve and -ve label from the dataset. I run 10-fold ...
1
vote
0answers
42 views

Sizing of training and validation sets in machine learning: Is there a proven optimum, or merely heuristics?

When I watch presentations where machine learning algorithms were used, the amount of data put in the training and validation sets seems to be somewhat arbitrary. Sometimes it's 80-20, sometimes it's ...
1
vote
2answers
93 views

How to report a SVM model to a 3rd party after cross-validation?

I have a binary classification problem. I trained my dataset using a Support Vector Machine (SVM). Now I want to report the model I trained to a 3rd party so that they can use. For the primal probem ...
3
votes
2answers
323 views

k folds cross validation on a multi-class dataset

Cross validation is one of the most important tools because it gives us an honest assessment of the true accuracy of our system. In other words, the cross-validation process provides a much more ...
4
votes
3answers
243 views

Can you compare different clustering methods on a dataset with no ground truth by cross-validation?

Currently, I am trying to analyze a text document dataset that has no ground truth. I was told that you can use k-fold cross validation to compare different clustering methods. However, the examples I ...
4
votes
3answers
333 views

Is a lower training accuracy possible in overfitting (one class SVM)

I am using the heart_scale data from LibSVM. The original data includes 13 features, but I only used 2 of them in order to plot the distributions in a figure. Instead of training the binary ...
0
votes
1answer
45 views

How to compare features and classifiers which achieve perfect accuracy?

So I'm looking to compare different combinations of features and classifiers. But I'm getting a lot of combinations that achieve 100% cross validation accuracy. I'm trying to figure out how I would ...
3
votes
3answers
52 views

How to compare features and classifiers which achieve perfect accuracy?

So I'm looking to compare different combinations of features and classifiers. But I'm getting a lot of combinations that achieve 100% cross validation accuracy. I'm trying to figure out how I would ...
0
votes
0answers
33 views

Problem with classifier prediction results

I built a classifier with 13 features ( no binary ones ) and normalized individually for each sample using scikit tool ( Normalizer().transform). When I make predictions it predicts all training sets ...