1
vote
0answers
18 views

Compare averaged GLM with boosted regression trees using cross validation : d2 and RMSE calculation

I want to compare BRT and averaged glm models on test sets by calculating the explained deviance and RMSE. How can I calculate d2 and RMSE from predictions? I use the following functions: gbm1 ...
1
vote
1answer
30 views

How to implement a hold-out validation in R

Let's say I'm using the Sonar data and I'd like to make a hold-out validation in R. I partitioned the data using the createFolds ...
0
votes
1answer
48 views

How can I perform 10-fold cross validation by manually constructing datasets?

I am working in text classification in RapidMiner where, because of the nature of my problem, I cannot use the built-in k-fold cross validation strategy, so I decided to create 10 copies of my dataset ...
0
votes
1answer
29 views

Grid Search for hyperparameter and feature selection

So I need to select my hyperparameters and also my features. A full grid search of the space of hyperparameters and features is too computationally intensive, so what I am doing instead is for each ...
1
vote
0answers
23 views

How to determine appropriate number of features and also which features to select?

So I have a dataset which I am using K fold cross validation on to select the number of features and which features should be selected. As I understand it, I would set the number of features to be ...
1
vote
0answers
27 views

What is the meaning of the term “enrichment” when performing cross-validation?

Trying to understand a discussion of a 5-fold cross-validation process to validate a predictive model and its results, there is a particular phrase which has me stumped, i.e.: The predictions of ...
1
vote
1answer
52 views

How is AUC of decision tree calculated?

I have a dataset which only has one continuous variable, and I try to use decision tree algorithm to build a model which classify the +ve and -ve label from the dataset. I run 10-fold ...
1
vote
0answers
31 views

Sizing of training and validation sets in machine learning: Is there a proven optimum, or merely heuristics?

When I watch presentations where machine learning algorithms were used, the amount of data put in the training and validation sets seems to be somewhat arbitrary. Sometimes it's 80-20, sometimes it's ...
1
vote
2answers
78 views

How to report a SVM model to a 3rd party after cross-validation?

I have a binary classification problem. I trained my dataset using a Support Vector Machine (SVM). Now I want to report the model I trained to a 3rd party so that they can use. For the primal probem ...
3
votes
2answers
113 views

k folds cross validation on a multi-class dataset

Cross validation is one of the most important tools because it gives us an honest assessment of the true accuracy of our system. In other words, the cross-validation process provides a much more ...
4
votes
3answers
148 views

Can you compare different clustering methods on a dataset with no ground truth by cross-validation?

Currently, I am trying to analyze a text document dataset that has no ground truth. I was told that you can use k-fold cross validation to compare different clustering methods. However, the examples I ...
4
votes
3answers
189 views

Is a lower training accuracy possible in overfitting (one class SVM)

I am using the heart_scale data from LibSVM. The original data includes 13 features, but I only used 2 of them in order to plot the distributions in a figure. Instead of training the binary ...
0
votes
1answer
36 views

How to compare features and classifiers which achieve perfect accuracy?

So I'm looking to compare different combinations of features and classifiers. But I'm getting a lot of combinations that achieve 100% cross validation accuracy. I'm trying to figure out how I would ...
2
votes
3answers
21 views

How to compare features and classifiers which achieve perfect accuracy?

So I'm looking to compare different combinations of features and classifiers. But I'm getting a lot of combinations that achieve 100% cross validation accuracy. I'm trying to figure out how I would ...
0
votes
0answers
29 views

Problem with classifier prediction results

I built a classifier with 13 features ( no binary ones ) and normalized individually for each sample using scikit tool ( Normalizer().transform). When I make predictions it predicts all training sets ...
5
votes
2answers
131 views

Why is k-fold cross validation a better idea than k-times resampling true validation?

I'm currently working through a machine learning textbook and just read a bit about k-fold cross validation, and I am wondering the following. I want to estimate a parameter, e.g. a penalty parameter ...
0
votes
0answers
73 views

Understanding the RandomForest with 10x10 cross validation for classification

I'm trying to understand the built of a random forest with the $10\times10$ cross validation for a binary classification problem. Therefore I have 4 basic questions: Notation: $N=500$ trees $i=$ ...
3
votes
1answer
172 views

How to choose train/test sample ratio, for machine learning?

I am building a real time machine learning module, which is not based on a huge** sample size, with hyper parameter grid search and cross validation process. I am looking for any insight/advice, as ...
9
votes
3answers
348 views

Cross-validation including training, validation, and testing. Why do we need three subsets?

I have a question regarding the Cross-validation process. I am in the middle of a course of the Machine Learning on the Cursera. One of the topic is about the Cross-validation. I found it slightly ...
0
votes
1answer
123 views

Feature selection and cross validation

I'm working on a project and I would like to know if the following strategy is good/correct. Sorry if this is a basic/stupid idea (I'm new to this). The input is a dataset with 2.500 features and ...
0
votes
0answers
44 views

Model selection for unbalanced data

How to do model selection for unbalanced data? how many data points from the whole data set should be selected for model selection? how many for training and testing?
2
votes
2answers
104 views

How is Hyndman's explanation of proper Time Series Cross Validation different from Leave-One-Out?

Hyndman's great explanation of proper time series CV is at the bottom of the page in the following link: http://robjhyndman.com/hyndsight/crossvalidation/ Leave-One-Out illustration in the following ...
0
votes
1answer
107 views

How do you do time series cross-validation using python? [closed]

Also, any tutorials/blogs available that you are aware of?
1
vote
1answer
201 views

Data split into training and test

I am implementing an EEG classifier with 15 subjects (patients), specifically a support vector machine classifier. I randomly choose the training and testing sets, but I was faced by a question "how ...
1
vote
1answer
314 views

Correct use of cross validation in LibsSVM

I am classifying data points from two different groups using LibSVM. I do the following: Creating the input file for LibSVM. ...
1
vote
1answer
98 views

Lower classification rate than expected by chance

I'm using scikit-learn for a small sample (36) classification problem with three features and three outputs (one output is binary and the other two are ternary). I'm using separate classifiers for ...
7
votes
3answers
656 views

Cross-validation or bootstrapping to evaluate classification performance?

What is the most appropriate sampling method to evaluate the performance of a classifier on a particular data set and compare it with other classifiers? Cross-validation seems to be standard practice, ...
2
votes
1answer
348 views

Does caret train function for glmnet cross-validate for both alpha and lambda?

Does the R caret package cross-validate over both alpha and lambda for the ...
0
votes
1answer
146 views

K-Fold Cross validation and F1 Measure Score for Document Retrieval using TF-IDF weighting and some customised weighting schemes

I am developing a search engine system based on the vector space model, and I am confused on what approach I should take to evaluate the system. My case is this: I have a set of indexed documents ...
2
votes
1answer
46 views

Dealing with few instances in support vector regression

What is the best way of going about dealing with few instances in support vector regression, e.g. only approximately 40? Also - is there an optimal way of dealing with outliers in this case of few ...
3
votes
1answer
401 views

R/caret: train and test sets vs. cross-validation?

This may be perhaps a silly question, but when generating a model with caret and using something like LOOCV or (even more to the point) ...
2
votes
1answer
275 views

Correct setup for leave-one-subject-out cross-validation

I've got a question concerning leave-one-subject-out cross-validation of a classifier and correct outlier handling in this case. Let's suppose I've got 5 subjects. Within each subject the features ...
1
vote
0answers
60 views

Does a less sparse matrix give less accurate estimation using cross-validation?

I am using a simple regression example $$Y=X\beta+\epsilon$$ $X$: 10 by 15 data matrix. $\beta$: 15 by 1 vector. $Y$: 10 by 1 vector. I am using beta1 and beta2 to compare the results against ...
8
votes
1answer
260 views

Model variance and bias in cross validation

This question is partly inspired about the answer to this other question: Number of folds for K-fold. The fundamental question I have is the following: How do different cross-validation methods ...
2
votes
0answers
85 views

Measuring parameter sensitivity and variability (standard-error) in k-fold cross-validation

I mainly use k-fold cross-validation for parameter tuning and model selection for prediction problems. Now, is there a standard or if not a less-known way to measure the sensitivity of the parameters ...
1
vote
0answers
311 views

k-fold cross validation vs k times hold-out validation

I am facing the evaluation of a genetic programming algorithm. I am using the Proben1 cancer1 dataset to evaluate the models created by this algorithm. This dataset contains 699 samples, which is ...
3
votes
1answer
184 views

Is it ok to determine early stopping using the validation set in 10-fold cross-validation?

I am working on a machine learning experiment comparing the use of multiple different neural network classifiers by applying them on a large number of datasets, using stratified 10-fold ...
3
votes
1answer
436 views

PCA before train/test split

I have a dataset for which I have multiple sets of binary labels. For each set of labels, I train a classifier, evaluating it by cross-validation. I want to reduce dimensionality using PCA. My ...
4
votes
5answers
817 views

How do you decide what your train, validation and test percentages are?

When splitting up my labeled data into training, validation and test sets, I have heard everything from 50/25/25 to 85/5/10. I am sure this depends on how you are going to use your model and how ...
1
vote
1answer
61 views

Inconsistency in cross-validation results

I have a set of dataset recorded from subjects as they perform some particular cognitive task. The data consists of 16 channels and a number of sample points per channel and I want to classify this ...
3
votes
1answer
150 views

Graphical representation of cross-validation errors for regression

What are some good ways of presenting/comparing cross-validated RMSE errors for regression using various models, graphically via plots? As of now, I have been presenting the quantitative results in ...
4
votes
1answer
329 views

How can you detect if a Gaussian process is over-fitting?

I am training a Gaussian process with an ARD kernel with lots of parameters by maximizing the marginal lielihood of the data, instead of cross-validation. I suspect that it is over-fitting. How can ...
1
vote
0answers
38 views

On cross-validation schemes for “rectangular” samples

Consider this example. Suppose that for any pair $(x, y)$ of bacterial strain $x$ and (candidate) anti-bacterial agent $y$, we can experimentally determine some measure $f(x, y)$ (say, the ...
2
votes
2answers
340 views

Evaluation method when using a large training set and a small test set

I am facing the evaluation of two text classifiers. I have a large training dataset (to be used for training only), and a separated small test set (to be used for testing only), both being balanced. ...
5
votes
5answers
857 views

Is using the same data for feature selection and cross-validation biased or not?

We have a small dataset (about 250 samples * 100 features) on which we want to build a binary classifier after selecting the best feature subset. Lets say that we partition the data into: Training, ...
4
votes
0answers
419 views

High-dimensional Regression Datasets [closed]

Am looking for pointers to publicly(online) available high-dimensional regression datasets for evaluating my research work. By high-dimensional, am looking for regression datsets with the number of ...
0
votes
1answer
195 views

Trying to run statistical tests in R but struggling as I am new to the language

Good Day, I believe this issue is more of a lack of understand of R (as I have never used it till recently) than anything else. What I am looking for is references, or documents to help me solve my ...
5
votes
2answers
2k views

libsvm “reaching max number of iterations” warning and cross-validation

I'm using libsvm in C-SVC mode with a polynomial kernel of degree 2 and I'm required to train multiple SVMs. Each training set has 10 features and 5000 vectors. During training, I am getting this ...
1
vote
1answer
246 views

Cross validation accuracy is the same as the fraction of negative labels - what does it mean?

I have a dataset for classification (binary - 1/0) that has around 4000 samples that I use to train the model (I'm using an SVM, if that's relevant). To check whether everything is working fine, I ...
1
vote
0answers
29 views

Evaluation and Testsets for NNMF

I am trying to evaluate my recommender system which uses Non-negative Matrix Factorization. Some things that I evaluate are How does the size of the feature matrix affect the recommendations How ...