Refers to general procedures that attempt to determine the generalizability of a statistical result. Cross-validation arises frequently in the context of assessing how a particular model fit predicts future observations. Methods for cross-validation usually involve withholding a random subset of the ...

learn more… | top users | synonyms

0
votes
0answers
5 views

Should I perform parameter tuning on the balanced or imbalanced dataset?

Consider a binary classification problem. As far as I know, if the dataset is imbalanced and if the two classification errors are not equally serious, then we should balance the distribution of the ...
1
vote
1answer
29 views

Computing Cross-Validation Errors for Subset Selection: question about standard code in the literature

I am currently trying to understand how to use cross-validation in order to choose among the "best" subsets of different sizes returned by the R function regsubsets (regsubsets returns the "best" ...
0
votes
0answers
12 views

Quantum computing and resampling techniques

Maybe I miss interpreted how does quantum computing work. If I understood well it would allow to perform extreme parallelization by making using a single qubit to perform many calculations at the ...
0
votes
0answers
5 views

Cross validation in link prediction

I'm attempting to predict what links will form in a large, sparse network, $G(V,E)$ and I can't figure out if I should evaluate on a training set or the full set. My predictor is simply looking at the ...
0
votes
0answers
8 views

cross validation GAM in Program R

I have performed a cross validation on a GAM model and have the following output: eval = CVgam(formula = time1 ~ strata(bout) + s(UNOBSTRUCTED_CHANNEL_WIDTH_FT),data = dat, nfold = 55, debug.level ...
0
votes
0answers
30 views

I have problem installing packages pbkrtest', 'BradleyTerry2', 'car', 'caret in R version 3.2.2) [on hold]

I have problem installing pbkrtest', 'BradleyTerry2', 'car', 'caret packages. I received the following warning; Installing package into ‘/home/kazem/R/x86_64-pc-linux-gnu-library/3.2’ (as ‘lib’ is ...
3
votes
2answers
53 views

Does cross-validation on simple or multiple linear regression make sense?

Does it make sense to apply train-test split or k-fold cross-validation to a simple linear regression model or multiple linear regression model? I'm really confused about this because I saw this ...
0
votes
0answers
9 views

SMOTE sampling does massively worsens results of Naive Bayes compared to up or down sampling

I train Naive Bayes (NB) and and artificial neural network (ANN) an imbalanced multiclass problem. In order to deal with the imbalance I resample the data set. Using 10-fold cv, the kappa statistics ...
0
votes
1answer
53 views

logistic regression predictive modeling

I would like to use a logistic regression for estimating the parameters of the logit function by using the maximum likelihood estimate. This amounts to minimizing the log-loss function, instead of ...
0
votes
0answers
20 views

The meaning of “training accuracy”?

If I split my data set into testing, training (further separated into subtraining and validation data set in cross-validation). In the context of machine learning and esp. in those ROC comparing the ...
0
votes
0answers
7 views

Combining Training and Validation Sets before passing model to Test Set

I have a situation in which I am building a model to predict a physical parameter of aircrafts, let's say it's parameter Y, Given several other measured parameters of aircrafts, let's call those ...
2
votes
0answers
27 views

How to correctly build a training/validation/test?

I have a database that is comprised of 3 sets of data. The first created in office conditions, the second in the same office taken on different day and the third drawn from random images from the ...
7
votes
3answers
772 views

Is cross-validation enough to prevent overfitting?

If I have a data, and I run a classification (let's say random forest on this data) with cross validation (let's say 5-folds), could I conclude that there is no over fitting in my method?
3
votes
0answers
42 views

Cross validation or EM for selecting strength of the prior?

Often when I'm looking at bayesian analyses, the influence of the prior is chosen via cross validation. For example, suppose $X$ and $Y$ represent some real valued data that I want perform a bayesian ...
3
votes
0answers
16 views

Method(s) to avoid overfitting model parameters for pre-determined model structure?

I'm using maximum likelihood estimation to fit a model of a pre-determined form to some data. To test this fitting method, I decided to generate some simulated data using the precise model form and ...
1
vote
0answers
13 views

Choosing alpha for cost complexity pruning as described in Introduction to Statistical Learning

In the following lectures Tree Methods, they describe a tree algorithm for cost complexity pruning on page 21. It says we apply cost complexity pruning to the large tree in order to obtain a sequence ...
1
vote
1answer
36 views

Random forest, cross validation or out-of-bag error?

I am training a random forest on a text data set (that I represent with synthetic features) and I am willing to assess the quality of the features I am creating. So far, I focused on the out-of-bag ...
0
votes
1answer
23 views

Evaluating fitlm (linear model) in matlab on a separate test set

I've built a linear model with Matlab using fitlm to predict the next value in a series of doubles. Now I would like to test this model on a different dataset so I get accuracy, p-value etc. I've ...
0
votes
1answer
35 views

Regression: find the best degree of polynomial with the best regularization parameter

When trying to predict data using linear regression or classify with logistic regression, with a polynomial, I know how to find the best degree of a polynomial to fits given data when the ...
0
votes
0answers
17 views

10-fold cross validation of a Multinomial Regression Model SPSS 20.0

I have a set of 125 people that belong to one of four nominal categories. Each person is described with 7 descriptors with 2-5 nominal variables that I use in my regression model to predict the ...
0
votes
0answers
16 views

Valid procedure for binary classification with cross validation

I have inherited a classification model for a binary parameter and have been asked if estimates can be improved. From this model, an equation has been put into some software for predicting this ...
1
vote
1answer
23 views

Modelling the distribution of rare events. How to validate?

I have been asked to create a model for the distribution of the number of items any previous customer might buy from a shop. Each customer will get a distribution of their own based of their past ...
0
votes
0answers
13 views

Assessing significance / confidence of a crossvalidated performance measure

I have a prediction model $P$ and I use some performance measure $I$ to measure $P$'s accuracy. The distribution of $I$ is unknown (it's a custom metric, which is somehow similar to the precision ...
0
votes
3answers
30 views

How to perform grid search effectively for tuning SVM parameters in cross validation?

I have C and gamma parameters for RBF kernel to perform SVM classification through cross validation in R software. How to fix values for grid search to tune C and gamma parameters? For example I took ...
0
votes
1answer
36 views

How to interpret my learning curve

I created the following learning curve in order to diagnose my Random Forest model. As I can see the curve indicates high variance and 'underfitting' (not overfitting), because cross-validation ...
1
vote
1answer
92 views

Variable selection using cross-validated PLS model when permutation test shows lack of significance

I understand that the permutation test on PLS can help to detect overfitting of the PLS model. Usually if the p-value is greater than a criterion, say 0.05, it means that the model is overfitting and ...
0
votes
1answer
19 views

Test accuracy is lower than the validate accuracy in classification

I separated my data set into three parts includes training, validate, and testing. I performed k-fold validation with using the validate set, then test the true performance of the predictive model ...
0
votes
0answers
18 views

Trying out a Leave-one-out cross validation and I'm not sure what to do with the output

I'm very new to cross validation, but I'm submitting some regression models as part of a paper and the reviewers are requesting it. I used the method listed here on a regression model with 200 ...
0
votes
2answers
70 views

choosing between logistic and discriminant

I am looking at regularized logistic regression, (l1 and l2 at the moment) and regularized discriminant analysis. How do I compare the two? I was thinking of doing gcv on both methods over a set of ...
0
votes
0answers
25 views

Out of Sample Cross Validation - Accuracy and Confusion Results

I have a scenario where I validate a trained model on an out of sample set - such that I begin by splitting the entire data set to train/test set. X_train, y_train, X_test, y_test. Then use ...
0
votes
0answers
29 views

Why is the “training score” I get from the learning curve of Multinomial Naive Bayes so different from the training score of the Bernoulli version?

I'm comparing the learning curves of Bernoulli and Multinomial Naive Bayes using the 20_newsgroups dataset from scikit-learn for text-classification. I considered both the "training score" and the ...
0
votes
0answers
5 views

Cross Validation using Zelig package [migrated]

I am using Zelig package inside R and i want to run cross-validation. ...
0
votes
0answers
47 views

Leave-one-out cross validation output interpretation and ROC curve

I have taken plenty of time to try and help myself, but I keep reaching dead ends. I have a dataset consisting of body measurements collected from a bird species, and the sex of each bird (known by ...
0
votes
0answers
8 views

Nested cross validation vs. split dataset into train, validation and test for parameter selection and performance evaluation

The goal is to get the unbiased performance estimation of the 'algorithm' (or model), e.g. precision and recall. And get a final model for practical usage. From what I read online, nested cross ...
0
votes
2answers
36 views

Predicting with cross validation

I want to predict labels via naive bayes and cross validation and measure the test accuracy. I do understand the principle of cross validation but not completely how to apply it. My question: Do I ...
0
votes
1answer
32 views

Is it correct to perform parameter tuning after nested cross validation?

Suppose that we have 3 different regression/classification methods: $f_1(D,\alpha)$, $f_2(D,\alpha)$, $f_3(D,\alpha)$ (for instance: lasso, neural network and SVM) where $D$ is the dataset and ...
1
vote
0answers
14 views

Data sampling for EFA, CFA, SEM, and beyond

Assuming I split my dataset (n = 650), for the purpose of performing exploratory factor analysis on half of the data, and then confirming the extractor factor structure using confirmatory factor ...
0
votes
0answers
20 views

Analysis of wrapper feature selection ouptput in Weka

I am using Weka to select important features from a dataset. I am using the wrapper method in this application. I chose a decision tree (j.48) for my classifier and Genetic search for the search ...
3
votes
2answers
64 views

Cross validated penalized logistic regression - one standard deviation rule

I am new to this topic and would like to understand it better. I want to build a binary classifier based on penalized logistic regression. I have 10 features and 23 observations: 16 from class "0" and ...
0
votes
0answers
9 views

Reason for Sci-kit DecisionTree very poor performance on UCI SoyBean compared to Weka J48

I am applying Sci-kit DecisionTreeClassifier classifier on the commonly used UCI soybean dataset. The resulting accuracy is only 0.49, which is very low compared to 0.93 I am getting for Weka's J48 on ...
1
vote
0answers
18 views

Standard deviation and CV

I'm using 10-fold CV to compute the classification accuracy of some classifier. I do the CV 10 times with randomly chosen folds. How do I correctly compute the standard deviation? Do I collect all ...
0
votes
1answer
83 views

Data Visualization after k-fold Cross Validation step

I need suggestion about data visualisation for cross validated data. I want to plot data after cross validation and need suggestions how to do that? I am thinking to plot like this If I use a ...
1
vote
1answer
25 views

Cross Validation and Nearest Neighbors

What is the best way to approach multiple-fold cross validation for a 1 nearest-neighbor model used for prediction? A common approach to cross validation is to, for example, split the dataset into ...
0
votes
0answers
15 views

Using cross validation to decrease variation in prediction outcomes

Im trying to predict the won category of some soccer matches. I have the following df: ...
0
votes
0answers
8 views

Score based training libraries

Suppose I have a machine learning problem. However, the evaluation of the function that we are trying to predict is quadratic weighted kappa and it cannot be evaluated on one single data point. It is ...
1
vote
2answers
36 views

Subset selection features acquired from randomized logistic regression

I learned about the concept of randomized logistic regression(or randomized lasso) recently. My data, biological data called Microarray, usually has large features but small samples - 10000 features ...
0
votes
1answer
32 views

Appropriate model for feature subset selection

I am working with a feature selection problem. What I am trying to do is find optimal subset of features for classification. My data consist of 100 features and 300 instances, and class label is ...
0
votes
0answers
19 views

Significance tests for streaming data

I want to compare multiple classifiers on multiple data streams. For a stream of length $n$ I test a single classifier each $t/c$ time steps using a dedicated (hold-out) subset of my data and ...
2
votes
1answer
93 views

Difference between bootstrap and resampling

I am using biological / microarray data. For example, one of my datasets has 50 samples, and 1000 gene attributes. They have 2 labels, Normal and ...
1
vote
1answer
58 views

Cross-validation vs random sampling for classification test

I usually have used cross-validation for testing classification performance. However, I read about the article that random sampling (bootstrapping) works better in many cases. I am not sure which one ...