# Tag Info

Accepted

### KNN: Should we randomly pick "folds" in RandomizedSearchCV?

As you say, the k-nearest neighbor algorithm has a number of hyperparameters such as the number of neighbors $K$ (but also e.g. distance function, how you summarize outcomes amongst neighbors etc.). ...
• 23.4k
Accepted

### Validation dataset is not a random sample from whole dataset

some of the sample in training will never show up in the validation set If I understand you correctly, you have samples (specimen, cases) of which you have multiple observations each. In that case, ...

### KNN: Should we randomly pick "folds" in RandomizedSearchCV?

I agree with your mentor that this is a bad idea. By randomly selecting the split that you will evaluate, you put the different actual hyperparameter values on unequal footings: maybe K=5 is best, ...
• 3,046

### overfitting of random forest in r

How does the 70 % accuracy of your CV compare to the rF's oob estimate? The behaviour you observe is to be expected for random forests, see also my old answer: https://stats.stackexchange.com/a/...
Accepted

### Why are sklearn's cross_val_score values not increasing with the size of the training set?

I don't think this result is too surprising. Each of the points in your plot has an associated error measurement associated with it. The overall number of holes only varies in a small range, so the ...
• 80.2k
Accepted

### How to use the Likelihood Ratio Test and Wald Statistic when also using Cross Validation?

Gradient descent (GD) is an optimization technique used for solving an optimization problem that yields parameter estimates of the model. It is a non-issue here, as GD is used in the same way in all ...
• 56.9k

### Choosing a Cut-Off Value from an ROC Curve for a Cross Validated Dataset

substantial variation in the cutoffs determined via cross validation tell you that your cutoff estimation algorithm does not yield stable results. However, that may just be because of small numbers ...
1 vote

### How should I train my CNN with a tiny dataset

You should start from a pre-trained model, replace it's output layer with a 3 class classification layer and finetune your model on your images. This is a standard procedure. Here's an example in ...
• 534
1 vote

### Is it possible to use cross-validation to estimate the reliability of a specific predictor?

Generally, cross-validation is used to assess predictive value. From that perspective, you could fit an otherwise identical model with and without a variable and assess the change in the root mean ...
1 vote

### Choosing a Cut-Off Value from an ROC Curve for a Cross Validated Dataset

Choosing cutoffs in general, and in particular based on indexes derived from retrospective sampling (ROC curve, sens, spec), is a process that is completely at odds with decision making, which is a ...
• 76.8k
1 vote

### Which data preprocessing steps do I need to perform on which data subset?

You have to make sure that your preprocessing of the training set doesn't use information from the validation or test set ("leakage"). E.g., in your k-nearest neighbor imputation, you must ...
• 6,248
1 vote

### overfitting of random forest in r

Random Forest Classifiers (RFC) with 100% training accuracy are not necessarily problematic. Make sure you are optimizing your hyperparameters on a separate validation set, this is especially ...
1 vote

### How likely is it that our model better than random in the upper corner of the AUC?

After you have built the model on the full original data set, you can estimate its generalizability via bootstrapping. The idea, under the bootstrap principle, is that repeated bootstrap sampling from ...
• 66.2k
1 vote

### Why are sklearn's cross_val_score values not increasing with the size of the training set?

To add to @sycorax' answer: If I understand the description of your data correctly, you have features: resistivity, density, ... (how many such physical properties do you have?) And in terms of ...

Only top scored, non community-wiki answers of a minimum length are eligible