Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
11 views

Using third validation set in Cross Validation?

(Note there's 2 paragraphs of background information before I get to the question) I've got a Neural Network classifier, trained with an EA to classify data. I previously used a holdout framework ...
0
votes
1answer
14 views

How to use k-fold cross validation in naive bayes classifier?

I'm trying to classify text using naive bayes classifier, and also want to use k-fold cross validation to validate the result of classification. But I'm still confused how to use the k-fold cross ...
1
vote
3answers
23 views

How to split dataset for time-series prediction?

I have historic sales data from a bakery (daily, over 3 years). Now I want to build a model to predict future sales (using features like weekday, weather variables, etc.). How should I split the ...
-3
votes
0answers
26 views

What should be the ms value for 10 fold cross validation [on hold]

I am trying to understand the predictive capability of my model using 10 fold cross-validation in R. I have total of 8880 data points. I got overall ms = 0.0767. I want to know what this value tells ...
0
votes
0answers
7 views

which is the right or better choice when doing k-fold cross validation for imbalance dataset

Supose we are faced a binary classification problem, where the class ratio of minority class to majority class is 1:5 in dataset. When we apply k-fold cross validation, which one of the following two ...
0
votes
0answers
7 views

Cross Validation and perfcurv in Matlab

I am trying to use perfcurv in a cross validation code. However at some point all the members of the test dataset are of the same class (0). My problem is a binary classification problem. Therefore ...
0
votes
0answers
8 views

CrossValidated KNN classifier

I am working with acoustic data. I just have 110 sound samples for training and test. So for solving the problem of having few test samples i want to use cross validation method. I am using KNN ...
0
votes
0answers
14 views

comparing models based on holdout method and n-fold CV

What to do when model choice based on n-fold CV results doesn't agree with holdout method results? I’m comparing ~100 models using both n-fold CV (jackknifing) and holdout method with expanding ...
0
votes
0answers
14 views

Feature selection when bagging trees/random forest

I want to get a better understanding of feature selection and how the number of features affect performance when bagging trees. I am using Matlab's treebagger and I ...
0
votes
0answers
15 views

cross validation for parameter-tuning a metaheuristic

For a certain problem, I've come up with a novel metaheuristic. The question I'd like to answer is "Does my metaheuristic perform better than previous methods over most problem instances?". My ...
0
votes
1answer
6 views

Select the most confident variable that has two features

Suppose now I have a group of students, and for each student two measurements are given: one is the height of the student and the other is the weight of the student. Then my question is how I can ...
1
vote
0answers
41 views

disadvantages of Neural network method

Hello Dear Researchers! I want to list the advantages and disadvantages of Neural network methods for classification or estimation purposes. I have already found the advantages of NN method in many ...
1
vote
2answers
33 views

Varying LIBSVM predictions based on test series labels

So I have a pretty well testing SVC train series which puts me into the mid 80 percentile without outrageous C/g values. My current C value is 2.0 and gamma is 0.5. Good numbers across the range ...
0
votes
0answers
21 views

Complete Logistic Regression framework using K-Cross validation

I'm implementing a logistic regression model in a low event rate data. I have gone through many webpages (including stackoverflow, including my questions) but none answer or describe the end-to-end ...
1
vote
0answers
98 views

Ensuring exploratory study's validity with pseudo-simple random sampling

The context of my questions is as follows. I'm performing a cross-sectional secondary research study, involving open source software (OSS) projects. I collect data (information about the projects) ...
0
votes
0answers
9 views

What should be the fitness function while using Particle Swarm optimisation

I am using Particle Swarm Optimisation for optimising the parameters of a Neural network (for multi-class classification problem). But what should be the fitness function for it ? I have tried ...
0
votes
0answers
39 views

R- Improving linear regression fit

I am trying to construct a predictive model in R. I am using the glm() in R to fit the model. I am getting a very high residual error after fitting the model. My target values are in the range of ...
0
votes
0answers
26 views

R- Improving linear regression fit [closed]

I am trying to construct a predictive model in R. I am using the glm() in R to fit the model. I am getting a very high residual error after fitting the model. My target values are in the range of ...
1
vote
1answer
31 views

Working with few data examples

I have been asked often in some interview, that how we should proceed when we have less data examples(say 50 or 100). What considerations needs to be made while choosing any algorithm. few points ...
0
votes
0answers
19 views

Low Accuracy using online logistic regression in mahout

I am getting very low value of accuracy on running online logistic regression on standard iris data (150 records). public static void main(String args[]) throws IOException { ...
0
votes
0answers
9 views

cross validation for kmodes in r

I am using k-modes (link) from the KlaR library (link) to cluster text data. I am not sure how to determine predictive error and thus perform cross-validation. Here is the "toy" sample, lets use ...
1
vote
0answers
20 views

logistic regression- validation dataset

I am working on getting propensity of Households to buy a certain product, I have completed the training dataset for running proc logistic in SAS, my question is 1) My training dataset is a biased ...
1
vote
1answer
35 views

Saving each step of Backward selection in R

I'm trying to recreate Leo Breiman's work http://www.stat.washington.edu/courses/stat527/s13/readings/BreimanSpector_1992.pdf and I'm experiencing some major difficulties in R. I've made it that far ...
1
vote
0answers
21 views

Follow-on to “Training with the full dataset after cross-validation” - sequential parameter estimation

Background: Here is the background for the question, both the question itself and the answer given by Dikran Marsupial. Training with the full dataset after cross-validation? It asks about after ...
0
votes
1answer
32 views

Detecting a consistent pattern in a dataset via Decision Trees and cross-validation

Assume a classification problem where there are two classes and the aim is to detect a consistent pattern which successfully separates the input dataset regardless of how we divide it into training / ...
1
vote
1answer
47 views

Selecting most realistic C and g params after gridsearch

I just ran an extended SVC gridsearch in libsvm on about 9000 multi-dimensional vectors representing a time series. Here are the highest scoring results: ...
0
votes
0answers
16 views

Getting the log-likelihood per fold in LOO Cross Validation for HMM in Matlab

I am running Leave-one-out Cross validations for deciding the optimal number of hidden states in an HMM. At each iteration I am getting one model and with forward algorithm I estimate the probability ...
0
votes
1answer
39 views

why is there a huge difference existed in coefficient of determination obtained from 10-fold cross validation?

I'm using gradient boosting regression model (GBRT). To evaluate this model, I use 10-fold cross validation, in each of which I set same param and compute the coefficient of determination as a ...
1
vote
0answers
24 views

Cross-validation in multi-level model

Suppose I want to estimate the out-of-sample prediction error of a boosted regression model that has random intercepts and slops. There are $G$ groups and $N$ observations. If I want to estimate the ...
0
votes
0answers
37 views

K-fold cross-validation for testing model accuracy in MATLAB [migrated]

I'm having some trouble truly understanding what's going in MATLAB's built-in functions of cross-validation. My goal is to develop a model for binary classification and test its accuracy by using ...
0
votes
0answers
30 views

Best Validation check number for MATLAB neural network

I'm using 10-fold cross validation and patternent function for a binary classification problem in ...
0
votes
1answer
25 views

How to get Sub-Training and Sub-Test from cross validation in Caret

I am using Caret function I have divided my data into training(75%) and test (25%) sets. Now I am running 10-Fold CV on training data. When i fit following model train_control <- ...
0
votes
0answers
46 views

Cross validating logistic GAM using CVgam

I have an ecological data set, whereby sediment mud content (%) (i.e. the continuous explanatory variable) is thought to be explaining the spatial distribution (i.e. presence/absence) of various ...
0
votes
0answers
18 views

All neural network designs stop because of early stopping in MATLAB

I'm using patternnet for my binary classification in MATLAB and using ...
0
votes
0answers
9 views

probability life similarities in a population cross section

I have just read up on Bouchard's Minnesota Twins study that turned up some amazing identical similarities in the lives of some identical twins reared apart. Both had same jobs, married and divorced ...
0
votes
0answers
43 views

Best K in K-fold cross validation

I'm using K-fold cross validation technique for generating train,test and ...
1
vote
2answers
71 views

How to choose the training, cross-validation, and test set sizes for small sample-size data?

Assume I have a small sample size, e.g. N=100, and two classes. How should I choose the training, cross-validation, and test set sizes for machine learning? I would intuitively pick Training set ...
0
votes
0answers
23 views

A question about warning that performing k-fold CV with caret

I am trying to createFolds function in caret to use k-fold cross-validation in R. But I came across this warning: ...
2
votes
0answers
101 views

Finding best neural network structure and inputs using optimization algorithm and cross-validation

I'm using optimization algorithm to find best structure+inputs of a patternnet neural network in MATLAB R2014a using ...
0
votes
0answers
50 views

Four tricky time series questions with a “seasonal twist”

A ski-hotel has the most guests in the third quarter in every year (check the data below after the four questions). Can you answer these four questions (every year has 4 values, the first is quarter ...
0
votes
2answers
45 views

Test data results does not match with cross validation results

I'm confused with my data I'm currently playing with. I have a data set which holds 58 attributes in 10000 instances. Attributes are 56 float values typically within 0 to 1. Then there is nominal ...
0
votes
0answers
26 views

How to evaluate a trained model using parameters other than AUC in RapidMiner?

I am using RapidMiner to build predictive models trained and cross-validated by a set of medical data(65 cases. 18 attributes), I am now running trials by trying different combinations of learners and ...
1
vote
1answer
32 views

How can a distribution of cross-validated $R^2$ scores be used to determine whether one model is significantly better than another?

I have two models, A and B. I have performed 10-fold cross-validation on both of them, so now I have 10 $R^2$ scores for each. How can I determine whether one is significantly better than the other? ...
7
votes
0answers
85 views

Why does the scikit-learn bootstrap function resample the test set?

When using bootstrapping for model evaluation, I always thought the out-of-bag samples were directly used as a test set. However, this appears not to be the case for the scikit-learn bootstrap ...
0
votes
0answers
64 views

Random Forest - understanding k fold cross validation

I am trying to improve my data science knowledge by solving problems available on the internet. I am currently using the R package randomForest to classify the ...
0
votes
0answers
24 views

Choosing fold size for highly Imbalanced dataset + nested CV + svm

I am trying to classify a dataset with ~1000 points. 90/10 is the class ratio - super imbalanced. Here are the following steps I did: Use 20 relevant features from previous knowledge Remove highly ...
0
votes
0answers
57 views

R - Random Forest - Need help understanding the rfcv function

My name is Abhi. I am trying to teach myself data science by solving some of the problems available on the internet. My current data set has about 900 reccords & 10 features. I am trying to use ...
0
votes
1answer
77 views

High Standard Deviation for Leave one out cross-validation?

I am using the leave one out cross-validation technique to evaluate my model. If the prediction on the test sample is right the output is 1 otherwise 0. So I have a array of N samples with 0's and 1's ...
3
votes
1answer
75 views

Cross validation with nonparametric smoothing regressions

When I use regression models I like to explore functional relationships using nonparametric smoothing regression (e.g. generalized additive models, lowess, running line smoothers, etc.) before ...
0
votes
0answers
33 views

Triple nested cross validation

I have read several very informative posts including the link about the nested/double cross validation, which can determine (sub)optimal hyperparameter values as well as make an unbiased estimate of ...