Tagged Questions

Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
24 views

Forecasting a solar data using arima in R [on hold]

I have a solar data collected from a PV plant for the period 2009.the method for forecasting as suggested for me is to use a training set for instance from 01/01/2009 to 30/04/2009 and a test period ...
1
vote
1answer
63 views

How do you evaluate a generative model?

Evaluating a discriminative model is relatively easy: compare the predictions with ground truth, using cross-validation. Unfortunately this strategy can't be used for generative models. Surely this ...
0
votes
0answers
26 views

Is 100% accuracy using randomForest indicative of anything wrong?

I am getting a 100% accurate result on randomForest model in R for loan default data even when my training set and test set are completely non-overlapping. I am using abt 8 parameters/features for ...
0
votes
2answers
49 views

Split clustered data into calibration and validation sample (Cross validation)

I have a dataset with >800 cases ($n$) from >30 ($k$) different organisations (clustered data). The number of cases within each organisation differ (unbalanced data; e.g.: organisation 1 = 30 cases, ...
1
vote
1answer
38 views

R caret package - number of principal components when preprocessing using PCA

I am using the caret package in R for training of binary SVM classifiers. For reduction of features I am preprocessing with PCA using the built in feature [preProc=c("pca")] when calling train(). How ...
1
vote
0answers
14 views

Confidence intervals for the Log Loss metric for model comparison?

Quite a few Kaggle competitions have used or are using the Logarithmic Loss metric as the quality measure of a submission. I'm wondering if there are other ways besides N-fold cross-validation to ...
6
votes
2answers
156 views

R t.test … NOT significant anymore

I got very confused while looking at help examples of the t.test function ...
0
votes
0answers
52 views

Cross Validation to find Misclassification rate of Explanatory Variables

I am trying to create a function that will allow me to identify which explanatory variable (x) in a logistic regression of a data set has the lowest rate of error in predicting a response variable (y) ...
1
vote
2answers
62 views

How to do cross-validation when comparing different feature selection methods?

I am using SVM for a prediction task. My sample size is small, only N=140. Suppose I want to compare the prediction accuracy when using two different feature selection methods. Would it be better to: ...
1
vote
0answers
33 views

Interpreting output from cvFit(), understanding cross-validation in classification tree model

I am trying to understand how to interpret the output for cvFit(). The data is from UCI's ML repository. This is my model ...
0
votes
0answers
18 views

cvFit mean predicted error interpretation for nls models

I have been using the cvFit() function of the cvTools library in order to test my models (nls() ones), but I would like to know more precisely what the cvFit() returns to me when it's done. It only ...
1
vote
1answer
45 views

Glmnet Caret Package with small number of observations

I have a regression problem where I’m attempting to train a data set with 70 predictors, but only 35 observations with glmnet in the caret package. I’m trying to determine the best resampling method. ...
3
votes
2answers
85 views

cross-validation to predict distribution of errors on finite test sets

In one use of k-fold cross-validation for evaluating classifiers, one trains k models, each on n(k-1)/k examples, and tests each on n/k examples. The average accuracy on those k test sets of size n/k ...
3
votes
0answers
39 views

LASSO prediction model question

I am trying to create a prediction model with 33 predictors (brain metabolite levels in various regions) and 8 observations (cognitive test scores) with p>>n problem using LASSO in MATLAB (...
8
votes
1answer
222 views

Can we use leave one out mean and standard deviation to reveal the outliers?

Suppose I have normally distributed data. For each element of the data I want to check how many SDs it is away from the mean. There might be an outlier in the data (likely only one, but might be also ...
1
vote
0answers
14 views

Cross-validating a survival model with right censoring?

Is it sensible and possible to cross-validate a survival model? Does it depend on whether there is censoring? If not, why? If the answer depends on the model, then answer for common survival models ...
0
votes
0answers
7 views

How to account for different ratio of samples during training and detection using a support vector machine (svm)?

Consider the following object recognition case: Detection of objects in an image using a sliding window approach in combination with a svm model. During sliding window search using multiple scale ...
0
votes
0answers
16 views

How to find a good model for an object recognition case using a support vector machine (svm)?

Consider the following example of an object recognition case: I'm trying to detect objects in an image using histograms of oriented gradients (hog) features. The feature vector resulting from hog is ...
0
votes
0answers
22 views

How to CORRECT un-reliable and un-stability in the prediction results

Currently, I meet such questions when building Random Forest model using my data set. My full data set: X_lab: 839 * 469 and y_lab: 839 * 1 which is for all labelled data and X_unl: 20346 * 469 which ...
0
votes
0answers
10 views

filter feature selection output and cross validation

If I use a filter method for ranking the features like Relief. suppose I have 100 features with 1000 sample and I used cross validation 3-fold . therefore I have 3 ranks for may features . at the end ...
1
vote
0answers
20 views

Distinguishing between different notions of $R^2$

What is the distinction between $R^2_{pop}$ – the population R-squared $R^2_{out}$ – the out-of-sample R-squared $R^2_{c.v.}$ – the squared population cross-validity coefficient ? These ...
0
votes
1answer
46 views

What is cross validation?

Yes, I know that after we fit (train) a predictive model on a training dataset, we need to test the fitted model on the testing-dataset. The motivation behind this procedure is clear: If our model ...
2
votes
2answers
50 views

Using K-fold cross validation to select a model's parameters

I think I understand completely the concept of cross validation, but there is one aspect I've never seen detailed. Let's assume I have a logistic regression model with four parameters I want to train. ...
1
vote
2answers
67 views

Different results from several “passes” of Random Forest on same dataset

I've been playing around with the German Credit dataset available in Kuhn & Johnson's caret package for ...
0
votes
0answers
33 views

Loss Functions and Evaluation Metrics

Do you have to evaluate with the same (or equivalent) loss function for model selection purpose? Say you have bunch of models to select. A loss function of one model in training stage is different ...
0
votes
0answers
20 views

Why would a reasonable range of the regularization parameters $\lambda$ be up to the maximum eigenvalue of the kernel matrix?

I was wondering, how do you choose a reasonable range for the regularization parameter $\lambda$ for regularized least squares when doing k-fold cross validation? I was told that a reasonable range ...
3
votes
4answers
96 views

Sample selection algorithms to ensure that training & validation sets are representative

Currently, I am encountering a question, which is how to selection representative samples (training set and test set, even validation set) from the whole data set? I would like build a classification ...
1
vote
0answers
30 views

Cross validation with unequal sample size for the left out sets

I am trying to do cross validation on several (20) subsets of samples, which all have unequal sample size. I cannot subsample so that sizes are equal. Example: batch 1: 500 samples batch 2: 400 ...
1
vote
1answer
38 views

10-fold cross-validation (high variation)

I am using 10-fold validation method to validate my model. I am using CART model and my sample size $\approx$ 50. Features $\approx$ 9. The 10-fold validated accuracy (averages) is about 76%. However, ...
0
votes
0answers
15 views

cross-validate hierarchical model for binomial data that is often sparse

I have binomial data (e.g, 130 successes in 4000 trials). In many, if not most, cells of interest, there were few trials and thereby few successes (e.g., 0 successes in 35 trials, 1 successes in 18 ...
1
vote
1answer
31 views

How to perform Cross-Validation for glasso to select lambda in R

I am using glasso for variable selection. To get the best possible value of lambda cross validation is recommended. However, I am not able to find how to perform cross validation for glasso in R. ...
0
votes
1answer
32 views

Probabilistic importance value for Caret linear SVM classifier

In a linear SVM model, inside caret I would like to get the variable importance after recursive feature elimination, so according to the documentation: ...
-1
votes
1answer
37 views

How to choose the right model after k-fold cross validation is done?

I'm using naive bayes to classify tweet into three classes. and i want to use k-fold cross validation to predict the right model, but i'm confused how to choose the right model after k-fold validation ...
1
vote
0answers
29 views

R - Warnings when using cv.lm

I've tried to apply cv.lm function instead of my own script which performs K-Fold cross validation and though the results are matching the formulas and my own thing, I keep getting warnings when using ...
0
votes
0answers
24 views

How do I get a classification report for my cross validated scores using sklean

I am running a logistic regression model using sklearn with 2 classes (1 and 0). Here is my code: ...
0
votes
0answers
10 views

Standard errors for the CV error curve using the boot package

Does anybody know how to obtain the standard errors for the CV error curve using the boot package? I understand the boot package can compute the K-fold CV for a fitted model, but I'd like to know if ...
0
votes
0answers
11 views

Comparing training and test data before cross-validation

When creating a train and test set to be used for cross validation, is it standard practice (or worthwhile) to, say, run Welch's t-tests on each feature between the two data sets to ensure that they ...
0
votes
0answers
10 views

Average accuracy score for leave-one-out cross validation

I need to show that average accuracy score for LOOCV is less or equal to (Number of support vectors trained)/(Size of training set). I have no hint on how to even start this. Any ideas? Thank you a ...
2
votes
1answer
20 views

AIC: relative versus absolute predictive error

I've read two interpretations of Akaike's Information Criterion (AIC) that seem to be in conflict, and I was hoping that someone could help me understand how to reconcile them. Interpretation 1: ...
2
votes
1answer
21 views

Cross-validation for nonlinear models that are linear in the parameters

I'd like to know if it's correct to the CV function in the forecast R package (http://cran.r-project.org/web/packages/forecast/forecast.pdf) to cross-validate a nonlinear model that is linear in the ...
1
vote
0answers
51 views

Deviance measure in glmnet package

for my current reseach I'm using the Lasso method via the glmnet package in R on a binomial dependent variable. In glmnet the optimal lambda is found via cross-validation and the resulting models ...
0
votes
0answers
32 views

Which R packages offer the foldid (or simliar) parameter for cross-validation of group lasso?

My situation: small sample size: 116 binary outcome variable long list of explanatory variables: 50 (both continuous and categorical) explanatory variables did not come from the top of my head; ...
3
votes
1answer
56 views

Why use stratified cross validation? Why does this not damage variance related benefit?

I've been told that is beneficial to use stratified cross validation especially when response classes are unbalanced. If one purpose of cross-validation is to help account for the randomness of our ...
0
votes
0answers
37 views

Using third validation set in Cross Validation?

(Note there's 2 paragraphs of background information before I get to the question) I've got a Neural Network classifier, trained with an EA to classify data. I previously used a holdout framework ...
0
votes
1answer
106 views

How to use k-fold cross validation in naive bayes classifier?

I'm trying to classify text using naive bayes classifier, and also want to use k-fold cross validation to validate the result of classification. But I'm still confused how to use the k-fold cross ...
0
votes
1answer
12 views

how to estimate correlation between two categorical variables in a cross table form

I have two variables AAA and BBB AAA has categories: 1-10,11-20,21-30,31-40,41-50. ...
2
votes
3answers
45 views

How to split dataset for time-series prediction?

I have historic sales data from a bakery (daily, over 3 years). Now I want to build a model to predict future sales (using features like weekday, weather variables, etc.). How should I split the ...
0
votes
0answers
18 views

Cross Validation and perfcurv in Matlab

I am trying to use perfcurv in a cross validation code. However at some point all the members of the test dataset are of the same class (0). My problem is a binary classification problem. Therefore ...
0
votes
0answers
10 views

CrossValidated KNN classifier

I am working with acoustic data. I just have 110 sound samples for training and test. So for solving the problem of having few test samples i want to use cross validation method. I am using KNN ...
0
votes
0answers
19 views

comparing models based on holdout method and n-fold CV

What to do when model choice based on n-fold CV results doesn't agree with holdout method results? I’m comparing ~100 models using both n-fold CV (jackknifing) and holdout method with expanding ...