Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
2answers
19 views

Varying LIBSVM predictions based on test series labels

So I have a pretty well testing SVC train series which puts me into the mid 80 percentile without outrageous C/g values. My current C value is 2.0 and gamma is 0.5. Good numbers across the range ...
0
votes
0answers
11 views

Complete Logistic Regression framework using K-Cross validation

I'm implementing a logistic regression model in a low event rate data. I have gone through many webpages (including stackoverflow, including my questions) but none answer or describe the end-to-end ...
1
vote
0answers
75 views

Ensuring exploratory study's validity with pseudo-simple random sampling

The context of my questions is as follows. I'm performing a cross-sectional secondary research study, involving open source software (OSS) projects. I collect data (information about the projects) ...
0
votes
0answers
7 views

What should be the fitness function while using Particle Swarm optimisation

I am using Particle Swarm Optimisation for optimising the parameters of a Neural network (for multi-class classification problem). But what should be the fitness function for it ? I have tried ...
0
votes
0answers
35 views

R- Improving linear regression fit

I am trying to construct a predictive model in R. I am using the glm() in R to fit the model. I am getting a very high residual error after fitting the model. My target values are in the range of ...
1
vote
1answer
28 views

Working with few data examples

I have been asked often in some interview, that how we should proceed when we have less data examples(say 50 or 100). What considerations needs to be made while choosing any algorithm. few points ...
0
votes
0answers
14 views

Low Accuracy using online logistic regression in mahout

I am getting very low value of accuracy on running online logistic regression on standard iris data (150 records). public static void main(String args[]) throws IOException { ...
0
votes
0answers
9 views

cross validation for kmodes in r

I am using k-modes (link) from the KlaR library (link) to cluster text data. I am not sure how to determine predictive error and thus perform cross-validation. Here is the "toy" sample, lets use ...
1
vote
0answers
18 views

logistic regression- validation dataset

I am working on getting propensity of Households to buy a certain product, I have completed the training dataset for running proc logistic in SAS, my question is 1) My training dataset is a biased ...
1
vote
1answer
30 views

Saving each step of Backward selection in R

I'm trying to recreate Leo Breiman's work http://www.stat.washington.edu/courses/stat527/s13/readings/BreimanSpector_1992.pdf and I'm experiencing some major difficulties in R. I've made it that far ...
1
vote
0answers
19 views

Follow-on to “Training with the full dataset after cross-validation” - sequential parameter estimation

Background: Here is the background for the question, both the question itself and the answer given by Dikran Marsupial. Training with the full dataset after cross-validation? It asks about after ...
0
votes
1answer
30 views

Detecting a consistent pattern in a dataset via Decision Trees and cross-validation

Assume a classification problem where there are two classes and the aim is to detect a consistent pattern which successfully separates the input dataset regardless of how we divide it into training / ...
0
votes
1answer
38 views

Selecting most realistic C and g params after gridsearch

I just ran an extended SVC gridsearch in libsvm on about 9000 multi-dimensional vectors representing a time series. Here are the highest scoring results: ...
0
votes
0answers
14 views

Getting the log-likelihood per fold in LOO Cross Validation for HMM in Matlab

I am running Leave-one-out Cross validations for deciding the optimal number of hidden states in an HMM. At each iteration I am getting one model and with forward algorithm I estimate the probability ...
0
votes
1answer
37 views

why is there a huge difference existed in coefficient of determination obtained from 10-fold cross validation?

I'm using gradient boosting regression model (GBRT). To evaluate this model, I use 10-fold cross validation, in each of which I set same param and compute the coefficient of determination as a ...
1
vote
0answers
22 views

Cross-validation in multi-level model

Suppose I want to estimate the out-of-sample prediction error of a boosted regression model that has random intercepts and slops. There are $G$ groups and $N$ observations. If I want to estimate the ...
0
votes
0answers
37 views

K-fold cross-validation for testing model accuracy in MATLAB [migrated]

I'm having some trouble truly understanding what's going in MATLAB's built-in functions of cross-validation. My goal is to develop a model for binary classification and test its accuracy by using ...
0
votes
0answers
24 views

Best Validation check number for MATLAB neural network

I'm using 10-fold cross validation and patternent function for a binary classification problem in ...
0
votes
1answer
23 views

How to get Sub-Training and Sub-Test from cross validation in Caret

I am using Caret function I have divided my data into training(75%) and test (25%) sets. Now I am running 10-Fold CV on training data. When i fit following model train_control <- ...
0
votes
0answers
42 views

Cross validating logistic GAM using CVgam

I have an ecological data set, whereby sediment mud content (%) (i.e. the continuous explanatory variable) is thought to be explaining the spatial distribution (i.e. presence/absence) of various ...
0
votes
0answers
14 views

All neural network designs stop because of early stopping in MATLAB

I'm using patternnet for my binary classification in MATLAB and using ...
0
votes
0answers
9 views

probability life similarities in a population cross section

I have just read up on Bouchard's Minnesota Twins study that turned up some amazing identical similarities in the lives of some identical twins reared apart. Both had same jobs, married and divorced ...
0
votes
0answers
32 views

Best K in K-fold cross validation

I'm using K-fold cross validation technique for generating train,test and ...
1
vote
2answers
51 views

How to choose the training, cross-validation, and test set sizes for small sample-size data?

Assume I have a small sample size, e.g. N=100, and two classes. How should I choose the training, cross-validation, and test set sizes for machine learning? I would intuitively pick Training set ...
0
votes
0answers
23 views

A question about warning that performing k-fold CV with caret

I am trying to createFolds function in caret to use k-fold cross-validation in R. But I came across this warning: ...
2
votes
0answers
90 views

Finding best neural network structure and inputs using optimization algorithm and cross-validation

I'm using optimization algorithm to find best structure+inputs of a patternnet neural network in MATLAB R2014a using ...
0
votes
0answers
47 views

Four tricky time series questions with a “seasonal twist”

A ski-hotel has the most guests in the third quarter in every year (check the data below after the four questions). Can you answer these four questions (every year has 4 values, the first is quarter ...
0
votes
2answers
43 views

Test data results does not match with cross validation results

I'm confused with my data I'm currently playing with. I have a data set which holds 58 attributes in 10000 instances. Attributes are 56 float values typically within 0 to 1. Then there is nominal ...
0
votes
0answers
19 views

How to evaluate a trained model using parameters other than AUC in RapidMiner?

I am using RapidMiner to build predictive models trained and cross-validated by a set of medical data(65 cases. 18 attributes), I am now running trials by trying different combinations of learners and ...
1
vote
1answer
31 views

How can a distribution of cross-validated $R^2$ scores be used to determine whether one model is significantly better than another?

I have two models, A and B. I have performed 10-fold cross-validation on both of them, so now I have 10 $R^2$ scores for each. How can I determine whether one is significantly better than the other? ...
7
votes
0answers
79 views

Why does the scikit-learn bootstrap function resample the test set?

When using bootstrapping for model evaluation, I always thought the out-of-bag samples were directly used as a test set. However, this appears not to be the case for the scikit-learn bootstrap ...
0
votes
0answers
53 views

Random Forest - understanding k fold cross validation

I am trying to improve my data science knowledge by solving problems available on the internet. I am currently using the R package randomForest to classify the ...
0
votes
0answers
21 views

Choosing fold size for highly Imbalanced dataset + nested CV + svm

I am trying to classify a dataset with ~1000 points. 90/10 is the class ratio - super imbalanced. Here are the following steps I did: Use 20 relevant features from previous knowledge Remove highly ...
0
votes
0answers
41 views

R - Random Forest - Need help understanding the rfcv function

My name is Abhi. I am trying to teach myself data science by solving some of the problems available on the internet. My current data set has about 900 reccords & 10 features. I am trying to use ...
0
votes
1answer
75 views

High Standard Deviation for Leave one out cross-validation?

I am using the leave one out cross-validation technique to evaluate my model. If the prediction on the test sample is right the output is 1 otherwise 0. So I have a array of N samples with 0's and 1's ...
3
votes
1answer
75 views

Cross validation with nonparametric smoothing regressions

When I use regression models I like to explore functional relationships using nonparametric smoothing regression (e.g. generalized additive models, lowess, running line smoothers, etc.) before ...
0
votes
0answers
32 views

Triple nested cross validation

I have read several very informative posts including the link about the nested/double cross validation, which can determine (sub)optimal hyperparameter values as well as make an unbiased estimate of ...
0
votes
1answer
63 views

Linear model- Understanding performances on training and test sets

I have a small normalized data set, 30 observations and 18 Predictors. All are continuous and some variable are related. I ran linear regression on it using Weka. The model automatically dropped some ...
0
votes
0answers
29 views

Validation accuracy larger than training accuracy

I was performing an experiment but got a higher validation accuracy than training accuracy. I've got a 39 mice data and performed leave one out cross-validation. The validation accuracy was 100%. But ...
2
votes
1answer
58 views

Cross-validation with dummy variables?

Does it make sense to use cross-validation with factor variables that have 3+ levels? When using bestglm, I get an error saying that it doesn't work with categorical variables. In the documentation ...
0
votes
0answers
17 views

How to forecast with quantile regressoin

If you have three quantile regression models with taus of 0.25, 0.5 and 0.75 and their coefficients how do you use these models to forecast a set of data not used to calculate the coefficients. In ...
0
votes
0answers
36 views

Parameter optimization of SVM

Currently I am using SVM to perform some classification task. I use libSVM with Matlab interface. From the practical guide of SVM (Link), we know that there are two parameters need to be tuned, namely ...
0
votes
0answers
11 views

Weka: cross validation using blocks of related instances (leave one patient out)

I have a dataset that comprises several instances for different patients, with multiple instances per patient. I need to perform some classification tasks and I was using cross-validation, but this ...
1
vote
1answer
46 views

Does it make sense to calculate Q2 and R2 values on PLS-DA models?

Since PLS-DA is a computational technique which deals with outcomes expressed as a categorical variable (e.g. "Yellow","Brown","Black","Green") I cannot understand how it is possible to calculate Q2 ...
2
votes
2answers
45 views

Does it make sense to do CV-error-weighted model averaging?

We often average models together to create an aggregate prediction model. Some recent research suggests that simple model averages perform as well or better than model averages weighted by functions ...
0
votes
1answer
23 views

how to impute missing values on numpy array created by train_test_split from pandas.DataFrame?

I'm working on the dataset with lots of NA values with sklearn and pandas.DataFrame. I implemented different imputation strategies for different columns of the dataFrame based column names. For ...
2
votes
2answers
53 views

Is cross validation for validating a model or for selecting best model in different kinds of models?

I am confused about the concept of cross validation and its usage. As I read about cross validation before, it is a way of validating a model. I did cross validation in my project (developing ...
1
vote
1answer
30 views

Can I use cross-validation to select optimal parameters SEPARATELY?

I'm wondering if there is any math/stat theory out there to support or deny this idea: I am using cross-validation and building models over a vector of parameter values to then choose the optimal ...
0
votes
0answers
29 views

Leave-one-out cross validation in selecting predictor

I am a newbie here. There are 155 total samples. Five different predictors Xi (i=1,2...5) are used to predict Y, like X1 X2 X3 X4 X5 Y .... The objective is to find the best predictor Xi to ...
0
votes
1answer
33 views

Caret: customizing feature selection, nested inside cross validation

Using caret, I want to train a SVM classifier and estimate its performance using repeated cross validation. My dataset has a very large number of predictors (300K) and I want to reduce this number ...