Tagged Questions
1
vote
0answers
65 views
k-fold cross validation vs k times hold-out validation
I am facing the evaluation of a genetic programming algorithm. I am using the Proben1 cancer1 dataset to evaluate the models created by this algorithm. This dataset contains 699 samples, which is ...
3
votes
1answer
61 views
Is it ok to determine early stopping using the validation set in 10-fold cross-validation?
I am working on a machine learning experiment comparing the use of multiple different neural network classifiers by applying them on a large number of datasets, using stratified 10-fold ...
3
votes
1answer
71 views
PCA before train/test split
I have a dataset for which I have multiple sets of binary labels. For each set of labels, I train a classifier, evaluating it by cross-validation.
I want to reduce dimensionality using PCA. My ...
3
votes
5answers
138 views
How do you decide what your train, validation and test percentages are?
When splitting up my labeled data into training, validation and test sets, I have heard everything from 50/25/25 to 85/5/10. I am sure this depends on how you are going to use your model and how ...
1
vote
1answer
43 views
Inconsistency in cross-validation results
I have a set of dataset recorded from subjects as they perform some particular cognitive task. The data consists of 16 channels and a number of sample points per channel and I want to classify this ...
3
votes
1answer
97 views
Graphical representation of cross-validation errors for regression
What are some good ways of presenting/comparing cross-validated RMSE errors for regression using various models, graphically via plots? As of now, I have been presenting the quantitative results in ...
1
vote
1answer
161 views
How can you detect if a Gaussian process is over-fitting?
I am training a Gaussian process with an ARD kernel with lots of parameters by maximizing the marginal lielihood of the data, instead of cross-validation.
I suspect that it is over-fitting. How can ...
1
vote
0answers
32 views
On cross-validation schemes for “rectangular” samples
Consider this example. Suppose that for any pair $(x, y)$ of bacterial strain $x$ and (candidate) anti-bacterial agent $y$, we can experimentally determine some measure $f(x, y)$ (say, the ...
2
votes
2answers
229 views
Evaluation method when using a large training set and a small test set
I am facing the evaluation of two text classifiers. I have a large training dataset (to be used for training only), and a separated small test set (to be used for testing only), both being balanced. ...
4
votes
5answers
368 views
Is using the same data for feature selection and cross-validation biased or not?
We have a small dataset (about 250 samples * 100 features) on which we want to build a binary classifier after selecting the best feature subset. Lets say that we partition the data into:
Training, ...
3
votes
0answers
278 views
High-dimensional Regression Datasets [closed]
Am looking for pointers to publicly(online) available high-dimensional regression datasets for evaluating my research work.
By high-dimensional, am looking for regression datsets with the number of ...
0
votes
1answer
183 views
Trying to run statistical tests in R but struggling as I am new to the language
Good Day, I believe this issue is more of a lack of understand of R (as I have never used it till recently) than anything else. What I am looking for is references, or documents to help me solve my ...
2
votes
1answer
720 views
libsvm “reaching max number of iterations” warning and cross-validation
I'm using libsvm in C-SVC mode with a polynomial kernel of degree 2 and I'm required to train multiple SVMs. Each training set has 10 features and 5000 vectors. During training, I am getting this ...
1
vote
1answer
177 views
Cross validation accuracy is the same as the fraction of negative labels - what does it mean?
I have a dataset for classification (binary - 1/0) that has around 4000 samples that I use to train the model (I'm using an SVM, if that's relevant). To check whether everything is working fine, I ...
1
vote
0answers
24 views
Evaluation and Testsets for NNMF
I am trying to evaluate my recommender system which uses Non-negative Matrix Factorization. Some things that I evaluate are
How does the size of the feature matrix affect the recommendations
How ...
2
votes
1answer
68 views
Is a different CV arrangement the same as a validation set?
I have a smallish dataset ~ 1500 rows X 500 columns. I've been using a standard 5 fold CV setup where row 1 = CV set1, row2 = CV set2, ... row 6 = CV set1,etc.
I'm at the point where I'm trying to ...
2
votes
1answer
468 views
k-fold cross-validation strategy for large data set in statistical learning
I'm trying to learn the Bayesian network structure from a very large data set, and the R package I used for learning can only handle a very small portion of the data set (~10%) at one time due to the ...
6
votes
3answers
991 views
Grid search on k-fold cross validation
I've a dataset of 120 samples in a 10-fold cross validation setting.
Currently, I pick the training data of the first holdout and do a 5-fold cross-validation on it to pick the values of gamma and C ...
3
votes
1answer
134 views
Using random sampling, is it better for the samples to be disjoint?
Background
I would like to measure the performance of a model trained on 3k samples, because this number of samples might be feasible to obtain in practice. I have a larger set of samples to choose ...
1
vote
0answers
73 views
How to use multiple datasets in order to measure the performance of a learning system?
I’m working on a project where I need to test a machine learning system which has a lot of hyper-parameters. Further, in order to gauge the performance of system, I’m planning to use several ...
4
votes
1answer
120 views
Estimating the variance of a bootstrap aggregator performance?
When performing cross-validation or bootstrap re sampling to estimate the performance of some machine learning algorithm, one commonly records the mean and variance of the errors obtained in of all ...
4
votes
2answers
430 views
Cross validation and prediction for unknown data
How do we build a model, cross validate it and use it to predict for unknown data?
Say I have a known dataset of 100 points. Steps for 10 fold cross-validation are-
Divide the data randomly into ...
12
votes
2answers
710 views
Variance estimates in k-fold cross-validation
K-fold cross-validation can be used to estimate the generalization capability of a given classifier. Can I (or should I) also compute a pooled variance from all validation runs in order to obtain a ...
2
votes
2answers
282 views
What are acceptable validation or cross validation error rates?
Is there a commonly acceptable error rate for validation? As in, if the error rate is less than X %, then my machine learning method would be considered "successful".
I'm looking for something ...
5
votes
2answers
311 views
Is cross-validation an effective approach for feature/model selection for microarray data?
I've been working with WEKA to build class predictors using this (rather old..) breast cancer dataset. The dataset is divided into a training and a test set. I've been testing different learning ...
13
votes
5answers
574 views
Can you overfit by training machine learning algorithms using CV/Bootstrap?
This question may well be too open ended to get a definitive answer, but hopefully not.
Machine learning algorithms, such as SVM, GBM, Random Forest etc, generally have some free parameters that, ...
15
votes
2answers
2k views
Choice of K in K-Fold cross validation
I've been using the K-Fold cross validation a few times now to evaluate performance of some learning algorithms, but I've always been puzzled as to how I should choose the value of K.
I've often seen ...
3
votes
2answers
685 views
Why does Lasso do better than SVM?
This is a soft-question:
I have been evaluation various regression techniques over a regression dataset that I have. I am surprised by the fact that cross-validated RMSE of Lasso is better than SVM ...
1
vote
0answers
101 views
Robust Support Vector Regression - robust to outliers
I've been reading/looking around for literature on support vector regressions that are relatively robust to outliers. I understand that standard SVRs can be significantly influenced by a few large ...
4
votes
2answers
400 views
Support vector regression on skewed/high kurtosis data
I'm using support vector regression to model some fairly skewed data (with high kurtosis). I've tried modeling the data directly but I'm getting erroneous predictions I think mainly due to the ...
1
vote
1answer
139 views
SVM parameter selection with NM simplex (or other algorithms)
I'm having some trouble getting the NM Simplex to find a good minimum for selecting hyperparameters of a rbf SVC. Not only am I tuning the 2 SVC parameters (C and gamma) I also have five class weights ...
2
votes
2answers
511 views
Grid search and tolerance in libsvm
I'm using libsvm and the 3-fold cross validation to select the best C and gamma, but I'm not sure for the range to use in the grid search. Is there any standard way to choose this range? I used: ...
2
votes
1answer
329 views
Strange kernlab's relevance vector machine predictions
I am using a relevance vector machine as implemented in the kernlab-package in R, trained on a dataset with 360 continuous variables (features) and 60 examples (also continuous, so it's a relevance ...
0
votes
1answer
148 views
Parameter Search for SVM on the whole data
I am trying to implement SVM and i did my parameter selection(grid search) on the whole data and used the best values of C and gamma from that search to test on the testing data.
Sometimes, the ...
5
votes
1answer
2k views
SVM rbf kernel - heuristic method for estimating gamma
I read on this exchange a heuristic method of estimating gamma for the rbf kernel in SVMs. I was wondering if someone might be able to explain it to me in a little more detail? I believe you select ...
2
votes
1answer
204 views
Multiclass SVM + Ineffective X Validation, Time Series Prediction
I've recently run into an interesting and rather odd problem with cross validating a multiclass SVM that I can't figure out. Basically, I have a timeseries to predict and have created a dataset of ...
2
votes
0answers
319 views
Cross validation procedure - is this right?
Just want to check that I am performing my cross validation procedures right. I'm using a non-linear svm. I do a five fold cross validation (5 splits of test/train on my original training data) and ...
2
votes
1answer
123 views
SVM and cross validation with a minimum finding algorithm
Just a simple question on parameter selection for SVMs. If I use a minimum finding algorithm to find the optimal parameters for a set of data, how do I "average" the parameters over a set of cross ...
2
votes
2answers
609 views
SVM parameter selection and cross validation
Have a quick question about parameter selection for an SVM. I'm using a rbf kernel, so trying to optimize C and gamma. I have an example set of around 4500, about 700 features, and using 700 examples ...
19
votes
3answers
250 views
Why is there an asymmetry between the training step and evaluation step?
It is well-known, especially in natural language processing, that machine learning should proceed in two steps, a training step and an evaluation step, and they should use different data. Why is ...
0
votes
1answer
221 views
Why doesn't classifier accuracy increase linearly with k in k-fold validation?
I am experimenting with WEKA and I'm trying to understand the impact of k in cross-fold validation. It seems reasonable to me that the higher k is, the more accurate the classifier will be (with ...
18
votes
7answers
464 views
How can I help ensure testing data does not leak into training data?
Suppose we have someone building a predictive model, but that someone is not necessarily well-versed in proper statistical or machine learning principles. Maybe we are helping that person as they are ...
10
votes
1answer
431 views
Is cross validation a proper substitute for validation set?
In text classification, I have a training set with about 800 samples, and a test set with about 150 samples. The test set has never been used, and waiting to be used until the end.
I am using the ...
4
votes
1answer
262 views
How to compare the results of a leave-one-out strategy?
I tried to reproduce the experiments described in this paper and wanted to compare the output of my system with the one described in the article. I am looking for a statistical comparison of the ...
0
votes
0answers
143 views
What machine learning techniques are especially susceptible to “over-tuning” of their hyperparameters?
For example, the random forest algorithm is not especially susceptible to "over-tuning" because it has 1 hyperparameter, "mtry," and mtry typically does not have a large influence on the result of the ...
2
votes
1answer
89 views
Is it valid to assign observations partially to both test and train in n-fold cross validation?
I'm looking to use n-fold cross validation for selecting meta-parameters for fitting a model to a dataset. However, dropping observations entirely from the learning-set while fitting the model to each ...
2
votes
1answer
432 views
Statistically comparing classifiers using only confusion matrix (or average accuracies)
Is it possible to perform a statistical test to determine if one classifier is better than the other using only the confusion matrices of these classifiers?
What about the average accuracies from ...
17
votes
5answers
2k views
Training with the full dataset after cross-validation?
Is it always a good idea to train with the full dataset after k-fold cross-validation? Or is it better instead to stick with one of the models learned in one of the cross-validation splits for ...
4
votes
1answer
253 views
Information on how value of k in k-fold cross-validation affects resulting accuracies
I've been doing some Machine Learning, and have been using k-fold cross-validation to assess the generalisation performance of the algorithm. I've tried k-fold cross-validation with k = 5 and k = 200 ...
6
votes
2answers
890 views
How to get generalisation performance from nnet in R using k-fold cross-validation?
I'm doing some Machine Learning in R using the nnet package. I want to estimate the generalisation performance of my classifier by using k-fold cross-validation.
...
