New answers tagged cross-validation
1
We need to revisit the definition of k-fold cross-validation: train set is divided into k equal parts. Each part is used for validation once. Cross-validation error is the average of k validation errors:
$CV(\lambda) = \frac{1}{k}\sum\limits_{i=1}^{k}E_i(\lambda)$
(For more details on cross-validation, read Hastie & Tibshirani, 2009)
As such, the ...
2
No, it's not ok because you used your validation set(s) in hyper parameter optimization, e.g. probably for choosing the right K or distance metric for KNN algorirhm. In your toy example, validation set has lower success while test set has higher. It can happen, but the converse situation (without much gap of course if there is not overfitting) is more common ...
1
I think this is a good chance to use nested cross validation for model selection. If you are optimizing over hyper parameters at the same time you are doing model selection, you risk being too optimistic about out of sample error. This part of the sklearn docs does a good job of explaining nested cross validation.
Fortunately, sklearn makes it really easy ...
0
As you've pointed out in your title - you're overfitting on the training set and your model is suffering from high variance. In general, you can address variance by:
Constraining the model
Using more training data
Resolving data quality issues and removing outliers
A clear starting point for variance reduction (the easiest thing to try), given the model's ...
1
In many situations: no. However, there are some steps you can take to a) ensure that what you're doing is sensible and b) push the limits a bit further.
The size of the validation set used to steer hyperparameter optimiziation directly influences the random uncertainty of the performance estimate you are trying to optimize.
Such an optimization usually ...
answered Dec 11 at 18:45
cbeleites supports Monica
27.9k2 gold badges59 silver badges112 bronze badges
2
If the validation set size is small, so as your training should be. Then you can perform cross-validation, even leave-one-out CV where the validation set is just one sample, assuming the training time is small. K-fold CV (including LOOCV) is typically more robust compared to using one constant validation set.
If you aren't able to do CV, performing searches ...
0
Context: The cross-validation method and the holdout method (train-test split) are seen as two methods to evaluate the model performance. The goal of this evaluation is to obtain an estimate of the generalization (or, test) error.
Summary: If the accuracy from the cross-validation method is less than the accuracy from the holdout method, it indicates model ...
1
I think there is some confusion as to the basis of what is being observed here. First, a model is trained against the X_train/Y_train dataset. When testing this model against the X_test/Y_test (holdout) dataset, an accuracy of 80-90% is observed. Next, a cross-validation was run. This outputs a fold score based on the X_train/Y_train dataset.
The question ...
0
Here is a kernel containing the a Group Stratified CV function:
https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation
This should allow to separate the folds while keeping constant proportion of the classes specified in some groups.
3
The answer to both questions is yes:
yes, LOO does have a pessimistic bias, and
yes, the described effect of additional pessimistic bias is well known.
Richard Hardy's answer gives a good explanation of the well-known slight pessimistic bias of a correctly performed resampling validation (including all flavors of cross validation).
However, the mechanism ...
answered Dec 9 at 14:28
cbeleites supports Monica
27.9k2 gold badges59 silver badges112 bronze badges
0
The paper suggested by @Fatima is very interesting and proposes multiple tests to pick the best model on a dataset.
Given that instead in this problem you have that you want to find the best dataset given the model, the approach should be slightly different because we are not in a "paired observations" test case, since the two datasets are different and ...
1
The following paper is one of the main literature that discussed statistical significance tests.
Dietterich, T. G. (1998). Approximate statistical tests for comparing supervised classification learning algorithms. Neural Computation, 10(7), 1895–1923.
https://sci2s.ugr.es/keel/pdf/algorithm/articulo/dietterich1998.pdf
1
This page of the documentation of scikit-learn has a pretty nice visual explanation of what are the differences between cross-validation sampling approaches. Here are some images for the methods you asked taken from the mentioned page.
As you can see, with KFold CV you divide the data in equal parts and pick train and test sets. For this method, I suggest ...
13
Given that the size of the training sample ($n_{training}$) is smaller than the size of the entire sample ($n$)
$$
n_{training}<n,
$$
the parameter estimates based on training subsamples in CV (be it LOO or K-fold) will in expectation be less accurate/precise than these based on the entire sample. This will cause the prediction loss from the model ...
0
To complete Adam Sampson answer, I should add that (not) oversampling the validation set is regard to the metric you use and the minority class.
If the POSITIVE in minority and you want to maximize the F1, recall, or precision, you do not need to oversample the validation set; but if NEGATIVE is minority (like you case with 266 NO), you may need to do ...
3
You aren't adding negative correlation correlation between observation and mean, you're taking out positive correlation between observation and mean. The whole problem with not doing cross-validation is that if you have n data points, then each time you do a prediction for one of the data points, 1/n of the prediction is coming from itself, so you're over ...
1
This blog talks about models being "y-aware". Essentially, anytime you use the outcomes to make a decision about the model, then that data can not be used in subsequent steps of model selection/development. Because the process you describe is essentially a form of hyperparameter optimization, then your model selection process is y-aware.
Therefore, your ...
11
This effect not only occurs in leave-one-out but k-fold cross-validation (CV) in general. Your training and your validation sets are not independent because any observation being allocated to your validation set obviously influences your training set (since it is being taken out from it).
To which extend this is the case depends on your data and predictor. ...
2
You need to check the accuracy difference between train and test set for each fold result. If your model gives you high training accuracy but low test accuracy so your model is overfitting. If your model does not give good training accuracy you can say your model is underfitting.
GridSearchCV is trying to find the best hyperparameters for your model. To do ...
1
Your approach is not wrong, though I would suggest a different interpretation. Those cases with higher probability values are ‘more likely to be defective, given the data’ rather than those cases are ‘most defective’. It is binary classification after all, and your training data either was or was not defective.
0
Cross validation, as you have seen, involves randomization. Therefore, any result derived from it will have some randomness. It is always good practice to repeat cross validation a couple of times (e.g., using different RNG seeds) to see how strong this randomness is.
If you have a small dataset, or a large model, your randomness will be larger than with a ...
answered Dec 2 at 13:56
S. Kolassa - Reinstate Monica
59.5k10 gold badges115 silver badges220 bronze badges
0
Choosing the best model is nothing but like hyper parameter optimization. We’re using training set to learn the parameters and validation set to learn the hyper parameters. In HPO, we typically evaluate the model on the candidate configurations and choose the best. In training, we use fancier stuff like gradient descent, adam optimizer etc. But still, they ...
0
The limited feature selection provided by LASSO (particularly with so few members of each class, a handful of features at most) would be unlikely to be very useful. The particular features selected would certainly be highly dependent on your particular data sample. Generalization to further cases could be poor.
You should instead take advantage of the many ...
2
From what I see in the current medical literature, both cross-validation or a validation based on a hold-out sample will likely get you published in OK, but not top journals (this may strongly depend on the standards in your sub-field, though). One caveat: If you are using a standard machine-learning approach (SVM etc.) you will have to perform hyper-...
2
Bias / error averaged across folds
In $k$-fold cross validation, you train $k$ times on a subset of $n \frac{k-1}{k}$ observations out of the original $n$. The remaining $\frac{n}{k}$ are tested.
Training on fewer cases will on average lead to somewhat worse performance unless the learning curve is flattened out for the training sample sizes we compare.
...
answered Nov 28 at 12:06
cbeleites supports Monica
27.9k2 gold badges59 silver badges112 bronze badges
1
I'm sorry to have given you the wrong impression. The optimal value of $\lambda$ obtained through cross-validation increases roughly with $\frac{p}{n}$. In the example of the linked question, the change is so large that you can expect $\lambda_\text{CV}$ to decrease. However, the rough part was perhaps understated in my answer there. The optimal penalty ...
1
If you look at a crosstab of the copy of allele variable I expect you will find the problem: There aren't enough people in some of the cells and you cannot estimate the parameters for those levels of that variable.
Also, it looks like you are using lm (which fits linear regression) but your DV (normal/abnormal) looks like it is categorical, which would ...
answered Nov 27 at 11:34
2
You are right to be cautious in the described situation, and also that
proper validation is needed.
However, splitting off a test set in addition to the cross validation would not be sufficient to get such a proper validation in the described situation.
I understand the situation where we perform cross validation. We keep a split of the data aside never ...
answered Nov 26 at 22:00
cbeleites supports Monica
27.9k2 gold badges59 silver badges112 bronze badges
0
Since there is an overlap between the training sets, the different folds of the cross-validation are not truly independent tests of the method. Nevertheless, sometimes that is the best you can do.
The difficulty here is that the training set is not very big. To get statistically robust estimate of the model's performance, you need a test set with hundreds ...
1
I'm aware this question is old but I landed here from Google anyway and the accepted answer isn't very pleasing as no one needs to programming CV themselves as this is handled by according libraries.
For a good answer first the scope terms must be defined. My answer focuses on machine learning ("classical" as in regression, random forest, etc... and not ...
1
You could do it, it is still OK. In the second one, shuffle option for KFold will be False, and you won't be able to set random_state in order for your analysis to be reproducible. Sometimes, it's more convenient to manage the splitter object at a greater detail, which is the case for the first usage.
1
I think your intuition is right.
The alternative is to perform multiple imputation on the entire dataset prior to splitting into train/test partitions. Doing so would mean that some information from the training sets is used to create/impute values in the test sets. In other words, there would be leakage from the training sets into the test sets, thereby ...
1
I'm not aware of any approaches that got to having their own name (other than that stratification is not per se restricted to classification).
Stratification is not per se restricted to classification.
Update: I just came across this paper: Xu et al.: Representative splitting cross validation, Chemometrics and Intelligent Laboratory Systems, 183 (2018) 29 -...
answered Nov 21 at 13:48
cbeleites supports Monica
27.9k2 gold badges59 silver badges112 bronze badges
2
There are many possible reasons, but it seems like you may not have enough rows to estimate the model accurately. You have enough degrees of freedom to vastly overfit, and because PLS regression finds the latent space that best models the covariance between the regressors and the target, it will find a space that overfits the data. As you expand the model ...
1
It is recommended to hold out a test set that the model only sees at the end, but not during the parameter tuning and model selection steps.
Grid search with cross-validation is especially useful to performs these steps, this is why the author only uses the train data.
If you use your whole data for this step, you will have picked a model and a parameter ...
0
After a bit of resaerch I found out that I can use pipeline in order to avoid data leakage.
references: this and this
This is how my code looks like now:
def produce_learning_curve(self, model, model_name, nb_splits, output_folder, parameters=None, nb_repeats=None):
X_train, y_train = self.X_train, self.y_train
pipe = Pipeline([
...
1
The key assumption of cross validation here is that the $k$ surrogate models are assumed to be (approximately) equal, and also (approximately) equal to the model trained on the whole data set.
These assumptions allow us to pool the test results from all $k$ folds and use them as approximation for the generalization performance of the model trained on the ...
answered Nov 18 at 10:53
cbeleites supports Monica
27.9k2 gold badges59 silver badges112 bronze badges
0
I agree with the OP that this terminology is awkward and confusing. Here's my take on it: native English speakers who are well-educated are used to terms such as "twofold" or "threefold", which sound just a bit antiquated but still usable. Critically, however, we don't see these words as containing the noun "fold"; "fold" is more of a suffix here, a funny ...
3
Your confusion might stem from the fact that for KNNs the workload shifts from training to predicting:
Training a KNN model means basically just loading the training data (as long as you do not do anything fancy like creating a hash table for more efficient neigbor lookup later). There is no optimization, no gradient descent, no weight adjustments etc. ...
1
Cross validation can be applied as long as the model is predictive (i.e. $\mathbf x \mapsto y$), regardless of how that model works internally. This general applicability is one of the strong advantages of cross validation. And it also implies that there is nothing special in the cross validation of a k-nearest neighbour model compared to, say, a logistic ...
answered Nov 17 at 12:50
cbeleites supports Monica
27.9k2 gold badges59 silver badges112 bronze badges
1
I understand you are talking about how to use early stopping to fit a final model. Well, it basically works the same as during model selection: you keep a small portion of the data as validation to select at which epoch to stop. So, in this case, you can't use the whole dataset to fit the final model.
Notice that model selection is used to select not only ...
0
RandomizedSearchCV discards the actual models it trains on each fold after evaluating them, so you won't be able to extract the fitted models from the output.
For refit=True, RandomizedSearchCV will pick the parameters that performed best on the validation sets and re-train the model with these parameters, this time on all observations. This is the model ...
Top 50 recent answers are included
