Repeatedly withholding subsets of the data during model fitting in order to quantify the model performance on the withheld data subsets.
0
votes
0answers
23 views
100% classification accuracy on validation dataset
I am trying to perform a multi-class classification where the network is trained to classify objects into 3 categories: cars, pedestrians and miscellaneous. I am using the KITTI Dataset for car ...
4
votes
1answer
13 views
KNN parameter tuning with cross validation: score draw
I'm trying to use the KNN method for binary classification. When trying to find the best 'k' parameter (the amount of neighbours that the algorithm looks at) I train a model on my training set and ...
1
vote
0answers
23 views
AIC based model selection, hyperparameter optimization and in-sample prediction
I'm using AIC to perform model selection along with hyperparameters optimization. The exact setup is the following:
I have two input variables (A and B), and a single target variable. All variables ...
0
votes
0answers
6 views
scikit learn nested cross validation: how to correctly standardize the outer cv data set?
I'm performing a nested cross validation where the inner loop is a GridSearchCV, which takes as estimator a Pipeline containing, among other things, a step for features standardization with ...
-1
votes
0answers
10 views
Workflow for nested crossvalidation that maximally utilizes caret
After reading this blog post by Max Kuhn, I successfully adapted the code as follows:
It now takes multiple tuning parameters for a given model.
Adding additional model types (e.g. KNN, ANN) only ...
3
votes
3answers
31 views
Cross validation : hyper-parameter tuning ? or model validation?
I have been seaching internet for exact definition of cross validation . I have come acrossed a few different ideas, with different terminology. I don't know if I have understand correctly.
Basically,...
1
vote
0answers
20 views
Cross validation and the Bias Variance trade-off
So I know that there have been a lot of questions about this topic but I try to understand it from a bit more theoretical/mathematical point of view.
I have some basic questions of how cross-...
2
votes
1answer
35 views
Is cross-validation better/worse than a third holdout set?
I see lots of papers that use just train and test datasets, without a third validation set, but they use cross-validation so that every data point is used for training and testing among the different ...
0
votes
0answers
6 views
10- fold cross-validation using Weka
I’m trying to figure out this problem with k-fold cross-validation.
A dataset about 76 booklovers shows some information (gender, age, number of books, likes Dan Brown, ..., bought your bestseller). ...
0
votes
0answers
11 views
Repeated CV evaluation with confidence intervals in R caret?
it occurs to me that there is a part of model evaluation that I have not understood yet. The problem that I am working on now illustrates the point well I think.
I need to fit a model of >400 ...
2
votes
1answer
27 views
1
vote
0answers
19 views
Expected Value of Cross Validation approximates Predictive Square Error
In the context of Smoothing Splines, Im trying to show that the expected value of the cross-validation can approximate the predictive square error. More specifically, I want to show that
$$E[(y_i - \...
0
votes
0answers
12 views
Is variable selection using cross validation prior to the selection of a class. method (eg. SVM, logistic regression, naive bayes, etc.) valid?
I plan on constructing a classifier using the following "algorithm":
Use the caret package in R to select variables
Train different classifiers using various methods on the variables found in (1) and ...
0
votes
0answers
6 views
Staticially comparing Error of two Distributions
I have a model that generates a probability distribution
Sum(p(1),...,p(n))=1
I have the ground truth, again
p'(1),...p'(n)
Their difference I defined as the error/variance.
I have a random ...
-1
votes
0answers
27 views
Unifying the validation set predictions in cross-validation?
This seems too good to be true, but I can't find any references that address it one way or the other. I'm assuming there's a problem with it, but I can't see it. Is there one?
Suppose we're using ...
0
votes
0answers
5 views
leave-one-out cross validation on images that have a discrete labels
How should I do leave-one-out cross validation on images that have a discrete labels (either Python or R)?
Most of the examples I see are quite different (they are not images).
0
votes
1answer
25 views
developing and assessing a prediction Cox model using lasso
I wonder if anyone can comment on if the following modelling strategy is valid please? I have a 200 patient survival data set (actually 2 data sets: 40 events and 160 events) and 100,000 ish candidate ...
0
votes
0answers
19 views
How does sklearn cross_val_score fit?
I'm trying to use cross validation to validate my results on the Pima Indians dataset. When I use sklearn's cross validation functions, the result it gives is no better than not training at all. I am ...
0
votes
0answers
21 views
Notation for Matrix Concatenation
I have an input data matrix $X$. In fact, I have $N$ of these input matrices, so I identify each as $X_i$ using an index $i \in \{1,\ldots,N\}$.
Thinking about something like cross validation, I want ...
0
votes
0answers
8 views
Determine the cutoff threshold for binary classification models using cross validation
I have a binary classification model which returns class probability scores as the outcome. I did the model parameter tuning using repeated k-fold cross-validation. I am wondering how can I decide the ...
1
vote
0answers
21 views
How is ShuffleSplit supposed to work with greedy feature selection?
I'm studying machine learning and wrote some code using the ShuffleSplit method in Python with linear regression using a greedy feature selection algorithm. However,...
0
votes
0answers
8 views
Why is one of the ROC curve generated by kfold always to the right?
Across multiple execution of this code:
...
0
votes
0answers
9 views
Group KFold for regression problems: When is it helpful?
I am working in manufacturing related industry. I am tasked with building some models to predict some physical quantities (for example, size of a hole in the structure) from spectra. The data usually ...
2
votes
1answer
60 views
How to meaningfully compute the accuracy of a multi-step forecast produced by a model
I am trying to measure the accuracy of my model in producing a multi-step forecast and I have read a lot of different opinions on the matter and am now rather confused.
The goal of my model is to ...
1
vote
2answers
61 views
Questions about performing k-fold cross validation for the first time
I am learning how to perform K-fold CV and I have a few questions about the method. This is an excerpt from the website: https://towardsdatascience.com/cross-validation-in-machine-learning-...
0
votes
0answers
25 views
Why is my ROC curve above the random line but the AUC is very low?
A little bit of background on my development process. I'm using a Random Forest model using the software package Alteryx (R Based) to classify a binary target variable that is approximately 60 (Neg - ...
0
votes
0answers
23 views
what is a valid way to compute out of sample - leave one out explained variance?
Say I have 30 samples and I compute a regression model in pseudocode:
...
0
votes
0answers
30 views
Results of cross validation don't consistent, what it mean
I have a data set which has about 100 samples. Each sample has 9 features ($x_1, ..., x_9$) and one targets ($y$). I tried ridge regression on this data set using sklearn. In the regression, the cost ...
0
votes
0answers
23 views
What is the equivalent of cross-validation for a fitted Cox Proportional Hazards model?
I am studying the effects of customer interaction on the probability that the customer will adopt a recommendation we make to them. For example, we might send out a first email, then a follow-up email ...
1
vote
0answers
17 views
What is the point of using PRESS instead of RMSECV?
What is the point of using predicted residual sum of squares (PRESS) instead of root-mean-squared-error-of-cross-validation(RMSECV)? In many books, especially in the area of chemometrics, the authors ...
2
votes
1answer
19 views
Confusion on the computation of Leave One Out cross validation?
1) I was studying about cross-validation and have a bit of confusion here. I understand about the k-fold technique, where if you ...
0
votes
0answers
7 views
Can adaptive learning rate method be used for dropout regularization?
if the neurons are deactivated randomly for each forward pass during an iteration, Can adaptive learning rate method for neural network such as RMSprop be used for the case of dropout regularization?
0
votes
1answer
19 views
Does dropout regularization prevent overfitting due to too many iterations?
For image classification problem, let's say, and given a neural network to train on,
if you were to run too many iterations for a single image of a cat would not generalize well into other images of ...
0
votes
0answers
15 views
How to perform leave one out validation on multivariate step-wise regression using PCA scores?
I am trying to create a multivariate regression model to predict Y using the scores from principle components analysis (PCA) done on some imaging data X which is composed of coordinates in 2-D space (...
0
votes
0answers
5 views
Compare the performance of a new model with the existing model
The challenge for me here is that I do not know the test/train datasets of the EXISTING model but I have the model with me (it is a logistic regression equation).
I believe the NEW model's AUC on ...
0
votes
0answers
13 views
Scikit-learn, cross-validation with parameter optimization
I wrote this idea in Python, but I'm confused as to the outcome of it.
I'm trying to do parameter tuning, with cross-validation. To do this, I have a classifier, a model selection strategy (...
1
vote
0answers
5 views
How to perform leave one subject out cross-validation for a two-level imbalanced data
How can I perform leave-one-subject-out (LOSO) cross-validation for a two-level imbalanced data. Specifically, I have a binary class problem, whereby the imbalance between two classes lies in: (1) the ...
2
votes
0answers
46 views
Leave-one-out cross validation for a linear model through the origin
An Introduction to Statistical Learning by James et al. defines the leave-one-out cross-validation estimate for least-squares as:
$$CV_{(n)}=\frac{1}{n}\sum_{i=1}^n \biggr(\frac{y_i-\hat y_i}{1-h_i}\...
0
votes
0answers
13 views
Compare accuracy between tools using k-fold cross validation, each tool is tested with different k values
I'm working on a new way to do the classification in a supervised way and I want to compare its accuracy to some related works. These works are using the same data set and they are testing their ...
4
votes
1answer
39 views
Does Leave One Out cross validation increase the chance of overfitting?
By increasing the size of the training set the model memorize more data. Thus, will using leave one out increase the chance of overfitting?
0
votes
0answers
36 views
Support Vector Machine Kernel Choice and hyper-parameter tuning for high class imbalance data
Thanks for your help in advance.
My question is this: Given the below information, is there some kernel preference and particular hyperparameter that is preferable to use when dealing with high class ...
0
votes
0answers
15 views
Multiple observations per subject - cross validation approach
I'm working on a modelling task to predict the probability of a customer becoming delinquent using gradient boosting. For each customer there are multiple observations including the borrower's ...
3
votes
0answers
47 views
Imputation and nested cross-validation
I am planning to do a nested cross-validation analysis using regularized regression. The inner loop will be used for model tuning and the outer loop for model assessment (test set). Because some data ...
0
votes
0answers
21 views
Cross validated model performance over different datasets
I am reading a paper which, for different datasets, repeated 5-fold cross validation 10 times on each one of them. For each dataset they calculated the mean of the metric and its standard deviation.
...
1
vote
0answers
25 views
Produce MSE by using cv.glm() in multiple linear regression, how about a transformed y variable
I applied 10-fold cross validation by using cv.glm() function in the linear regressions.
I am able to obtain the MSE in this way,
mse=cv.glm(data,model,K=10)$delta
However, if I applied ...
0
votes
1answer
42 views
How to prevent overfitting with regression using ranger (randomforest)
I use caret to train the model (on Boston dataset from the mlbench package).
Here is the code
...
0
votes
0answers
10 views
What kind of Statistic Method for enrichment or overrepresent should I used for a rank ordered vector with Binary status
I have a gene expression data from 1065 different cell lines, let's say "BRAF" gene. BRAF gene expression levels are ordered. Most TP53 mutated cell lines are high BRAF expression (see the figure ...
1
vote
2answers
64 views
Using the standard deviation in Cross Validation
I'm running a Grid Search to find the optimal parameters for xgboost via sklearn.
I can see that the grid search picks the set of parameters with lowest mean MSE.
The problem is that upon inspecting ...
0
votes
1answer
26 views
Unbalanced dataset accuracy [duplicate]
I’m currently encountering some problems analyzing a dataset with neural network. The problem is that I have an unbalanced binary class training set (10:1). Training accuracy for both classes are 100%....
2
votes
1answer
28 views
How do you cross-validate moving time predictions?
I have a model that trains itself by looking at the last 12 months of data, and then predicts the next month (out of sample).
Say I have 24 months worth of data, thus allowing me 12 opportunities to ...
