Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

1
vote
0answers
27 views

Gold Standard data for training but not validation

My goal is to determine which among the three classification algorithms perform better { Logist Reg or Neural Network or SVM }. I have a training dataset and the ...
0
votes
0answers
9 views

How to run a nested LabelKFold with scikit-learn? [on hold]

I have a dataset with ~300 points and 32 distinct labels and I want to evaluate a LinearSVR model by plotting its learning curve using grid search and LabelKFold validation. The code I have looks ...
0
votes
1answer
18 views

R e1071 SVM always gives me (in average) below changes cross validation accuracy with random data

I am running e1071 linear SVM on my neuroimaging data. ( by function svm() ) When I was doing permutation tests, I found, in average, the cross validation (CV) accuracies with shuffle labels were ...
1
vote
0answers
17 views

Sample dependency in Neural Net Training cross-validation

I've created a Monte Carlo simulation that randomly divides my data into "test" and "training"-Samples and then trains a neural network. The ratio of 0 and 1 (19.62%) Category is stabilized on ...
0
votes
0answers
6 views

Is Weka test options cross-validation and percentage split deterministic? [on hold]

If a have a dataset and use cross-validation-10 or percentage split 66%, it'll always yield the same data for my tests? Is the sampling stratified and random with a constant seed for the random ...
3
votes
0answers
31 views

K-fold cross-validation for time series with dynamic target variable (Scikit)

I would like to do a K-fold cross-validation on time series data (market data) with a two class classification target. My test folds must be forward looking and of a fixed size ...
0
votes
1answer
57 views

Feature/variable selection with categorical variables [closed]

My goal is to compare several machine learning algorithms for sales prediction (logistic regression, neural network, random forest, svm -> classification problem, whether the sales will go up or down) ...
0
votes
1answer
30 views

How do we obtain a final/best model when using k-fold cross-validation?

My past assumption of cross-validation (in particular k-fold CV) was that in order to given same chance to each sample in our dataset to appear in training , we use k-fold CV. Under my assumption we ...
0
votes
1answer
32 views

combining RMSE for multiple cross-validation procedures

I have implemented a leave one out cross validation to calculate errors between daily forecast and observed values for spatio-temporal data taken in a given season (summer say). I have further ...
1
vote
0answers
32 views

Spark K fold Cross validation [closed]

I’m having some trouble understanding Spark’s cross validation. Any example I have seen uses it for parameter tuning, but I assumed that it would just do regular K-fold cross validation as well? What ...
1
vote
2answers
53 views

Repeatedly split data in training (0.75) and test (0.25) for cross validation

What kind of cross validation is it called when we randomly split the data into 0.75 training and 0.25 test data set. And this split is done 1000 times.
19
votes
5answers
995 views

Overfitting: No silver bullet?

My understanding is that even when following proper cross validation and model selection procedures, overfitting will happen if one searches for a model hard enough, unless one imposes restrictions on ...
1
vote
0answers
43 views

Frisch Waugh theorem for removing fixed effect

My data have three levels: industry, firm, and year. I would like to find how much firm variation of sales is explained by age. I want to use the Frisch Waugh theorem. I follow the following steps: ...
1
vote
1answer
24 views

question regarding the process of feature selection, model building and k fold cross validation

I have a question regarding the process of feature selection, model building and k fold cross validation. I have a forty features and 200 records data sets. I want to select down to 10-12 features ...
2
votes
1answer
69 views

Find out if using k-fold cross-validation helped to overcome overfitting (Machine Learning standard)

One of the main way to overcome overfitting is using $K$ fold cross-validation, and as this paragraph in cross-validation wiki page says: The goal of cross-validation is to estimate the expected ...
0
votes
1answer
26 views

How does cross-validation in train (caret) precisely work?

I have read quite a number of posts on the caret package and I am specifically interested in the train function. However, I am not completely sure if I have understood correctly how the train function ...
0
votes
0answers
4 views

Cross-validation on XGBClassifier for multiclass classification in python [migrated]

I'm trying to perform cross-validation on a XGBClassifier for a multi-class classification problem using the following code adapted from http://www.analyticsvidhya.com/blog/2016/03/complete-guide-...
0
votes
2answers
83 views

Why is cross validation better than training once with the whole data set? [duplicate]

A form of cross validation takes multiple subsets of a data and trains the model on them, then tests on the remaining subsets. Eventually, all subsets will be trained on. Although I know that ...
1
vote
1answer
44 views

Cross-validated prediction error worse for LARS than ordinary linear regression

I am analysing microarray data in order to build a model for predicting cell proliferation (a continuous variable) based on gene expression (also a continuous variable). There are many more genes than ...
1
vote
1answer
17 views

How to choose the test set size when the training set size is given?

I have data on 64 subjects collected in a medical setting. With the help of ROC curve analysis and bootstrapping, I have identificed two predictors for illness(present or not present) in the group. ...
0
votes
1answer
72 views

Partitioning training data for dimension reduction and classification

Let's say I want to test the performance of my dimension reduction + classification pipeline. To do this, I will use k-fold cross validation. I know that performing dimension reduction on the complete ...
0
votes
1answer
37 views

Cross validation function from which programming language is more appropriate? [closed]

I'd like to use resample to achieve stable results of an unsupervised algorithm that finds clusters in data. I'll use k-fold cross validation repeated many times but I'm in doubt if I should use R or ...
0
votes
0answers
21 views

How to select the number of basis functions in functional data analysis? [closed]

In functional data analysis, I could not understand how to select the number of basis functions for a particular data. I followed the paper, where it is mentioned that there is no thumb rule to select ...
0
votes
0answers
11 views

choosing test set in a seasonal time series

The data is sequential, but not necessarily continuous, ie. there are multiple gaps between the start and end date. I fit a regression model, which may or may not involve lagged variables, and I want ...
5
votes
1answer
88 views

Overfitting when dealing with nearly the same number of features and observations

When dealing with nearly same number of features and observations, one of the most common problem is overfitting. For my project I used 2 class LDA on a 1400 * 1000 dataset and to avoid overfitting, ...
1
vote
0answers
9 views

What's the most appropriate way to derive and validate a model with hierarchical data

I am working on a model to predict the risk of some outcomes and could really use some advise: Let's say we have x number of patients, each patient have anywhere between 0 and y number of visits (...
0
votes
0answers
34 views

Variable selection in logistic regression model

Imagine a data set with approximately 100 variables and 5000 cases. The outcome is a two-level factor. All variables are factors, most of them three levels (yes, no, or indifferent). After building a ...
0
votes
0answers
11 views

How to use the dataset for leave-one-ut CV?

I would appreciate some feedback on my leave one out CV procedure, because I am not sure it works correctly. 1.Load the files I am using 26 binary classified articles. Files are shuffled when ...
1
vote
2answers
49 views

Does repeated k-fold cross-validation give the same answers each time?

I have $n$ instances in my data and I will do 5-fold cross validation on it (like in the picture): But when I read about "repeated cross-validation" I think that it will give me exactly the same ...
0
votes
0answers
7 views

Which will be better If I have 2 years data (training and testing) with a condition

Condition: I will always asks the model to predict the behavior of last 1 month data i.e I want the result on last 1 month of data. I have 2 years of data of my app, and I have to train the model and ...
1
vote
1answer
38 views

Why does k-fold cross validations help obtain stable clustering results?

Why applying k-fold cross validation helps obtaining stable clustering results in unsupervised learning? How is this done? Thanks
0
votes
0answers
18 views

How does gamma in SVM RBF kernel influence the accuracy?

I am working on a classification program using SVM RBF kernel. To find the best parameters C and gamma, I used grid search, and got the image below. What confuses me is that when gamma varies from 0.3 ...
1
vote
1answer
46 views

What is the right way to use SVM with cross validation?

I read a lots of discussions and articles and I am a bit confused on how to use SVM in the right way with cross-validation. If we consider 50 samples and 10 features describing them. First I split ...
0
votes
1answer
31 views

What are the alternatives to MMRE, PRED and MdMRE for validation?

I am working over the statistical validation of data. Till now I have computed MMRE, PRED and MdMRE. But I need alternatives to these because MRE is sensitive to data with large MRE's.
0
votes
0answers
26 views

What is the formula of Median Magnitude of Relative Error? (MdMRE)

I'm familiar with these terms 1 - MRE Mean Relative Error 2 - MMRE Mean Magnitude of Relative Error I need to compute MdMRE which is Median Mean Relative Error. I searched on the net but didn't ...
0
votes
0answers
14 views

Quick question on performance of lasso logisitc model

I performed a lasso logistic regression on two modles. One model contains only the control variables. The other model contains controls+linguistic measures. When I search for the optimum lambda ...
0
votes
1answer
33 views

Best approach for selecting averaging weights

I am trying to build an R tool for forecasting a (hopefully) wide range of time-series. I have settled on using several models, taking the forecasts from each, and deriving a weighed average of them ...
2
votes
0answers
49 views

Unstable projection in LDA space in $n<p$ situation [duplicate]

I'm trying to classify (LDA) few samples (n=12) in a high dimensional feature space (p=24) into 3 classes. First I reduced the ...
0
votes
0answers
40 views

Why does RMSE underestimate model variance?

I have read that RMSE of calibration/validation/cross validation is frequently used for model selection (e.g., for ANN), but can lead to over-fitting because the prediction error represents the ...
0
votes
0answers
9 views

Why am I getting the same value for F1 and accuracy?

I trained and SVM classifier and I noticed that I'm getting equal F1 and accuracy values (using a cross-validation), which means that the number of True-Positives and True-Negatives is the same. The ...
1
vote
1answer
29 views

Cross Validation Train Test Gap Question

Question: is minimizing test set mean validation error more important than the gap between train and test errors? Let's say I can tweak parameters in my model to give me mean validation error of 4500 ...
0
votes
0answers
10 views

How to split a survival data such that the proportion of events and censoring are equal in both groups

I need to develop a prognostic model, i have the survival data, and i need to split into validation set and training set. However, I want the Ratio of event to censoring in both sets to be equal. so ...
0
votes
1answer
51 views

Performance of regresion tree rpart

I am running a regression tree using rpart and I would like to understand how well it is performing. I know that rpart has cross validation built in, so I should not divide the dataset before of the ...
2
votes
2answers
36 views

k-fold crossvalidation and independent models evaluation

I have a question about using k-fold cross-validation. From what I have read, there are several steps for using it. Shuffle my data. Train my model for each fold on training data and test it using ...
0
votes
3answers
47 views

Finding the optimal value of k in the k-nearest-neighbor classifier: is this cross-validation?

I have collected 1000 data points with each data point belonging to eight categories. I would like to be able to correctly estimate the categories of any new data by using the k-nearest-neighbor ...
2
votes
0answers
24 views

Hierarchical Cluster Analysis validation

I have never used Hierarchical Cluster Analysis for inferential statistics before, but the dendrogram it produces provides a nice way to visualise my data. I applied the HCA to my variables with the <...
0
votes
0answers
37 views

How to perform multiclass SVM classification using k-fold cross-validation and SMO with some kernel method in MATLAB?

I have data matrix X.csv file of size nxd, where n are the observations, and d variables. There are c_1,..., c_m classes. Let, Y be the matrix containing the class labels. There is no header row in ...
0
votes
0answers
25 views

Using confidence intervals with Simple Linear Regression

So simple linear regression is performed on 3000 data points, and 1000 data points are withheld. How can we use confidence intervals, along with the withheld data points, to assess the predictive ...
0
votes
0answers
15 views

Cross validated loglikelihood?

This is probably a silly question: I was playing around with penalized package and cvl outputs a cross validated loglikelihood and another measure just called loglikelihood which is suppose to be "...
0
votes
0answers
39 views

Scaling the data in a decision tree changed my results?

I know that a decision tree doesn't get affected by scaling the data but when I scale the data within my decision tree it gives me a bad performance (bad recall, precision and accuracy) But when I ...