5
votes
Accepted
KNN: Should we randomly pick "folds" in RandomizedSearchCV?
As you say, the k-nearest neighbor algorithm has a number of hyperparameters such as the number of neighbors $K$ (but also e.g. distance function, how you summarize outcomes amongst neighbors etc.). ...
- 23.4k
3
votes
Accepted
Validation dataset is not a random sample from whole dataset
some of the sample in training will never show up in the validation set
If I understand you correctly, you have samples (specimen, cases) of which you have multiple observations each.
In that case, ...
3
votes
KNN: Should we randomly pick "folds" in RandomizedSearchCV?
I agree with your mentor that this is a bad idea. By randomly selecting the split that you will evaluate, you put the different actual hyperparameter values on unequal footings: maybe K=5 is best, ...
- 3,046
2
votes
overfitting of random forest in r
How does the 70 % accuracy of your CV compare to the rF's oob estimate?
The behaviour you observe is to be expected for random forests, see also my old answer: https://stats.stackexchange.com/a/...
2
votes
Accepted
Why are sklearn's cross_val_score values not increasing with the size of the training set?
I don't think this result is too surprising. Each of the points in your plot has an associated error measurement associated with it. The overall number of holes only varies in a small range, so the ...
- 80.2k
2
votes
Accepted
How to use the Likelihood Ratio Test and Wald Statistic when also using Cross Validation?
Gradient descent (GD) is an optimization technique used for solving an optimization problem that yields parameter estimates of the model. It is a non-issue here, as GD is used in the same way in all ...
- 56.9k
2
votes
Choosing a Cut-Off Value from an ROC Curve for a Cross Validated Dataset
substantial variation in the cutoffs determined via cross validation tell you that your cutoff estimation algorithm does not yield stable results.
However, that may just be because of small numbers ...
1
vote
How should I train my CNN with a tiny dataset
You should start from a pre-trained model, replace it's output layer with a 3 class classification layer and finetune your model on your images.
This is a standard procedure. Here's an example in ...
- 534
1
vote
Is it possible to use cross-validation to estimate the reliability of a specific predictor?
Generally, cross-validation is used to assess predictive value. From that perspective, you could fit an otherwise identical model with and without a variable and assess the change in the root mean ...
1
vote
Choosing a Cut-Off Value from an ROC Curve for a Cross Validated Dataset
Choosing cutoffs in general, and in particular based on indexes derived from retrospective sampling (ROC curve, sens, spec), is a process that is completely at odds with decision making, which is a ...
- 76.8k
1
vote
Which data preprocessing steps do I need to perform on which data subset?
You have to make sure that your preprocessing of the training set doesn't use information from the validation or test set ("leakage"). E.g., in your k-nearest neighbor imputation, you must ...
- 6,248
1
vote
overfitting of random forest in r
Random Forest Classifiers (RFC) with 100% training accuracy are not necessarily problematic.
Make sure you are optimizing your hyperparameters on a separate validation set, this is especially ...
- 63
1
vote
How likely is it that our model better than random in the upper corner of the AUC?
After you have built the model on the full original data set, you can estimate its generalizability via bootstrapping. The idea, under the bootstrap principle, is that repeated bootstrap sampling from ...
- 66.2k
1
vote
Why are sklearn's cross_val_score values not increasing with the size of the training set?
To add to @sycorax' answer:
If I understand the description of your data correctly, you have
features: resistivity, density, ...
(how many such physical properties do you have?)
And in terms of ...
Only top scored, non community-wiki answers of a minimum length are eligible