Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.

2
votes
0answers
15 views

Confidence intervals of AUC obtained by merging/pooling predictions from different test sets

I have one question regarding the CIs of the AUROC calculated merging/pooling the predictions coming from different test sets. In one analysis, we use a sort of nested cross-validation approach, ...
0
votes
0answers
7 views

Create data partitions based on a factor for use in caret::train() [on hold]

long time reader but first time poster. Please bear with me if I make a few mistakes! Here is a bit of context - I have almost 200 datasets of the same type of data all in the tidy data format. The ...
0
votes
0answers
11 views

Data Partition and Cross Validation

I'm curious, is it wise to do a manual data partition; say I have a 400 data and I part manually 360 data for training and the 40 rest for testing, and then use the acquired training weight (without ...
1
vote
2answers
33 views

Is it ok to do parameter tuning after cross validation?

I am thinking of using cross validation to select the best algorithm (e.g. SVMs, Random Forest), and then doing parameter tuning on the selected algorithm to build a model. Is it acceptable and how ...
0
votes
0answers
14 views
0
votes
1answer
48 views

When we should NOT use k-fold cross validation to assess the predictor?

Does anybody know in which cases --of learning and predicting--, it is better to use "validation test" or something else, instead of "k-fold cross validation" to assess the performance of the ...
1
vote
2answers
39 views

What fraction of the training set should I use? [on hold]

I tried to fit RandomForestClassifier with n_estimators=500 to the training dataset which has 600,000 instances and it's taking ...
0
votes
1answer
13 views

Robustness of ranking based on cross validation set, should I bootstrap?

I have a dataset that I separate into training and validation. I fit several models on the training set and then evaluate their performance on the validation set. When I rank the models based on their ...
0
votes
1answer
25 views

Machine Learning: Cross-validation score

In sklearn example http://scikit-learn.org/stable/tutorial/statistical_inference/model_selection.html Why we leave 2 trials to test but we can receive 0.93489148580968284 score. I guess we only have ...
1
vote
0answers
19 views

Is there a problem when the lower and upper limits of a confidence interval are the same?

I have done the bootstrap technique and I want to make inference. I have calculated the 95% confidence intervals, however, for one of the variables in my regression model its lower and upper limits ...
0
votes
0answers
12 views

Why sklearn call two time the score function when doing cross validation each step? [closed]

I'm using GridSearchCV with 3 folds and every time a step is performed, sklearn calls 2 times the function score with different ...
0
votes
0answers
10 views

Cross validation using external regressors

I'm quite new to forecasting. I'm trying to use the tsCV() command for an ARIMA model with external regressors. I'm failing in writing the right function to use in the tsCV() command My time series ...
0
votes
0answers
17 views

Whats the proper way to statistically compare stochastic classifiers?

Say I have 2 neural networks (or any other stochastic classifier) and I want to compare these methods. I can run 10-fold cross validation, and get a mean and standard deviation for each. I can then ...
1
vote
2answers
41 views

Techniques to avoid overfitting

I have heard of several techniques to avoid overfitting: Validation curve: which let us choose the set of parameter with the minimum step between validation score and training score. But it seems ...
0
votes
2answers
16 views

Plot ROC or PR curves from either the X,Y coordinates (i.e TPR/TNR; or PPV/TPR) or list of predictions (class probabilities)? [closed]

I have a list of X,Y coordinates for plotting both a ROC curve and a PR curve. I also have the data which was used to calculate those coordinates (i.e. a list of individual predictions with binary ...
0
votes
0answers
7 views

Nested cross validation with balanced down-sampled training data set - fairness and proper method?

Example data set: 150 positive examples (fold size = 15; 9 folds = 135). 1770 negative examples (fold size = 177; 9 folds = 1593). The problem: When performing ...
2
votes
1answer
31 views

How to validate the association rules or results obtained from Market Basket Analysis? Train-test methodology

If I have a large set of transactions where in each has a set of goods and I want to do market basket analysis (affinity analysis) using Apriori. However, compared to traditional supervised machine ...
2
votes
1answer
147 views

Need advice on evaluating forecast accuracy in R

I'm trying to evaluate some software for forecast accuracy. It works by summing up all the orders from a number of locations for each month, then determines the best model out of a series of models ...
0
votes
0answers
7 views

improve model roc_auc score

From the GridSearchcv on a random forest classifier, the best parameters is giving me an auc_roc score of 0.80. But when i train a new random forest model with the best parameters i am getting an ...
3
votes
5answers
359 views

Is using both training and test sets for hyperparameter tuning overfitting?

You have a training and a test set. You combine them and do something like GridSearch to decide the hyperparameters of the model. Then, you fit a model on the training set using these hyperparameters, ...
0
votes
0answers
18 views

Cross validation on train dataset

I am performing classification on a dataset of $20477$ samples. I have done a $70-30$ split and fit a randomForest model on the train data. I am getting an accuracy of 99% on the train data. On the ...
0
votes
1answer
35 views

How to build a ROC curve or PR curve from outer cross-validation predictions?

I have a dataset of 165 positive and 1700 negative. For training I balanced them to 132 positive (80%) and 132 negative (80% of minority class). For testing, I leave the default natural distribution ...
2
votes
0answers
23 views

How can I tell a model reached the optimal parameters?

Aside from stacking more models, If I want to know if I have arrived the best possible single model(the best parameter), is there anything/process I can tell? Assume I made n-degree of polynomial ...
0
votes
0answers
13 views

A good performance with n-fold cross validation and overfiting

Is it possible to train a model (for regression or classification) that has overfited bu that that has a good performance in a n-fold cross validation?
0
votes
0answers
26 views

Choosing a model

I am working on a sales forecast right now and I have created 4 models but I am unsure which one to use. I have 17 Quarters of data(4 Full years + 1 QTR) and I am only looking to forecast 2 quarters ...
0
votes
0answers
8 views

Discrepancy between coefficients generated by cv.glmnet() and glmnet()?

I noticed small differences between the coefficients generated by cv.glmnet() and glmnet() when the same lambda was applied. I am wondering why this happens. Codes below will reproduce the phenomenon ...
0
votes
1answer
37 views

How to decide moving window size for time series prediction?

I have a model to predict +1 day ahead of this time series. Looking at the chart you can notice some seasonality every 5 days. I suspect using a moving window as training set could help me making a ...
2
votes
1answer
27 views

Error and Dispersion meaning in tune.out for SVM Classifier

I am using a SVM to solve a binary classification problem with qualitative response as output. To find out the best parameters for the SVM I used a 10-fold cross-validation technique. And the result ...
0
votes
0answers
22 views

Overfitting, cross validation and validation curve

Some time ago i used to choose my hyperparameters only relying on the Test score returned by cross validation. But after reading How does cross-validation overcome the overfitting problem? i doubted ...
1
vote
0answers
21 views

Isn't HM a better way of averaging k fold cross validation scores than AM?

When calculating fscore we use the harmonic mean of precision and recall since hm penalizes situations when either of the two metrics is low while the other is large unlike the arithmetic mean. So ...
0
votes
0answers
11 views

Comparing the same classifier on paired data

I am using a Linear Support Vector Classifier with sklearn for a project of mine and evaluate the performance of my classifier with nested cross-validation (1:1:1 train:validation:test). In particular,...
1
vote
1answer
31 views

ValueError: Requesting 3-fold cross-validation but provided less than 3 examples for at least one class. Any way to get away with this?

I am trying to perform multiclass classification on a 10 class dataset with around 650 data points. But whenever trying to run the code, it gives the above-mentioned error. Although, I understand what ...
0
votes
3answers
68 views

How to find out if a model is overfitted? [duplicate]

I have built 2 models: 1) precision: 0.80 - AUC ROC: 0.69 2) precision: 0.90 - AUC ROC: 0.94 I have posted both them to Kaggle as Titanic competition, the first model scored 0.7 and the second one ...
0
votes
0answers
12 views

Cross-fold validation - f1 scores iterations (sudden drop)

I am performing a cross-fold validation in my model to find the best number of iterations to train the model based on f1 scores. For some fold iterations, I find that there is a sudden drop in the F1 ...
3
votes
1answer
107 views

Feature Distribution in Cross-Validation

In the case of binary classification, stratified cross-validation only ensures that each fold contains roughly the same proportions of the two types of class labels. When does it make sense to also ...
3
votes
1answer
39 views

Cross Validation and Confidence Interval of the True Error

I'm interested in the relation between Cross Validation and the True Error Estimation of a Classifier (Chapter 5 - Machine Learning - Mitchell). Suppose we have 150 examples, I decide to use a 100 ...
2
votes
2answers
115 views

Feature selection using cross validation

I am dealing with a typical $p > n$ problem in the medical field. (typically $p \approx 3700$ and $n \approx 100$ ). The dependent variable is binary (healthy/sick) and features are continuous ...
1
vote
1answer
38 views

K-Fold on a Random Forest

I have a 1400x120 matrix that I'm aiming to run a random forest regression on but have been running into difficulty understanding how the RF interacts with K-fold. Specifically, is cross validation ...
1
vote
0answers
24 views

How to do cross validation [closed]

I have a dataset with two parameters, LifeExpectancy, GDPercapita and I need to compare some regression linear modelswith Leave-One-Out Cross-Validation. (by MSE), first I need to make p regression ...
0
votes
1answer
37 views

Nadaraya Watson Bandwidth-Variance

I'm working with the Nadaraya Watson estimator and calculate the optimal bandwidth h with leave one out cross validation. Now I'd like to get the variance of h (not the variance of the NW estimator)....
1
vote
3answers
48 views

R2CV increases over R2

I have a dataset: X Y 4706 77983 7217 48187 5314 1098 10725 91683 10725 27366 And want to show that there is no correlation. ...
0
votes
1answer
26 views

Cross Validation in application of clustering on a collection of similarly behaving time series

I'm trying to understand how and at which point can one apply Cross Validation for time series data. If i'm not wrong CV increases generalisation so that our model has less bias in case the data is ...
1
vote
1answer
24 views

leave-one-out sensitivity analysis for regression based population study

Using a linear mixed model I observe significant associations, however, the coefficients are very subtle. To show that the model is not based on only a few observations I want to validate the model ...
0
votes
0answers
21 views

Model exploration and nested cross-validation?

When using nested cross-validation, can different models be explored in a principled way, without spoiling the final estimate of model performance? When evaluating a model, the data used to build or ...
0
votes
1answer
39 views

Validation set for early stopping

I learnt about validation for early stopping by taking the course, and I have several questions. Question about validation for early stopping: We split the dataset $D$ into a training set $G$ and a ...
0
votes
0answers
13 views

Sklearn / GridsearchCV: roc_auc score better with evaluating against accuracy than roc_auc

I've run into the following problem which is kinda puzzling me. I've two GridSearch classes configured, one with the scoring set to roc_auc and the other using ...
0
votes
0answers
23 views

Multiple Linear Regression Cross Validation - How does it work?

Say X is a linear combination of 3 predictors, A,B and C, plus some noise. For each of these variables I have 30 years of data. I can compute a multiple linear regression of A,B,C predicting X, which ...
0
votes
0answers
17 views

ARIMA Cross-Validation with Range of Orders

I have been working my way through Hyndman's Forecasting: Principles & Practice and tried to replicate Table 8.2, but instead of manually passing Arima() the ...
0
votes
0answers
19 views

Best practice for choosing hyperparameter in cross-validation

I have splitted my whole data into pairs of train set(80% of all data) and validation set(20%).In the end of process of training and validation I have to choose hyperparameter basing on train and ...