5 votes
Accepted

KNN: Should we randomly pick "folds" in RandomizedSearchCV?

As you say, the k-nearest neighbor algorithm has a number of hyperparameters such as the number of neighbors $K$ (but also e.g. distance function, how you summarize outcomes amongst neighbors etc.). ...
user avatar
  • 23.4k
3 votes
Accepted

Validation dataset is not a random sample from whole dataset

some of the sample in training will never show up in the validation set If I understand you correctly, you have samples (specimen, cases) of which you have multiple observations each. In that case, ...
user avatar
3 votes

KNN: Should we randomly pick "folds" in RandomizedSearchCV?

I agree with your mentor that this is a bad idea. By randomly selecting the split that you will evaluate, you put the different actual hyperparameter values on unequal footings: maybe K=5 is best, ...
user avatar
  • 3,046
2 votes

overfitting of random forest in r

How does the 70 % accuracy of your CV compare to the rF's oob estimate? The behaviour you observe is to be expected for random forests, see also my old answer: https://stats.stackexchange.com/a/...
user avatar
2 votes
Accepted

Why are sklearn's cross_val_score values not increasing with the size of the training set?

I don't think this result is too surprising. Each of the points in your plot has an associated error measurement associated with it. The overall number of holes only varies in a small range, so the ...
user avatar
  • 80.2k
2 votes
Accepted

How to use the Likelihood Ratio Test and Wald Statistic when also using Cross Validation?

Gradient descent (GD) is an optimization technique used for solving an optimization problem that yields parameter estimates of the model. It is a non-issue here, as GD is used in the same way in all ...
user avatar
2 votes

Choosing a Cut-Off Value from an ROC Curve for a Cross Validated Dataset

substantial variation in the cutoffs determined via cross validation tell you that your cutoff estimation algorithm does not yield stable results. However, that may just be because of small numbers ...
user avatar
1 vote

How should I train my CNN with a tiny dataset

You should start from a pre-trained model, replace it's output layer with a 3 class classification layer and finetune your model on your images. This is a standard procedure. Here's an example in ...
user avatar
  • 534
1 vote

Is it possible to use cross-validation to estimate the reliability of a specific predictor?

Generally, cross-validation is used to assess predictive value. From that perspective, you could fit an otherwise identical model with and without a variable and assess the change in the root mean ...
user avatar
1 vote

Choosing a Cut-Off Value from an ROC Curve for a Cross Validated Dataset

Choosing cutoffs in general, and in particular based on indexes derived from retrospective sampling (ROC curve, sens, spec), is a process that is completely at odds with decision making, which is a ...
user avatar
1 vote

Which data preprocessing steps do I need to perform on which data subset?

You have to make sure that your preprocessing of the training set doesn't use information from the validation or test set ("leakage"). E.g., in your k-nearest neighbor imputation, you must ...
user avatar
  • 6,248
1 vote

overfitting of random forest in r

Random Forest Classifiers (RFC) with 100% training accuracy are not necessarily problematic. Make sure you are optimizing your hyperparameters on a separate validation set, this is especially ...
user avatar
1 vote

How likely is it that our model better than random in the upper corner of the AUC?

After you have built the model on the full original data set, you can estimate its generalizability via bootstrapping. The idea, under the bootstrap principle, is that repeated bootstrap sampling from ...
user avatar
  • 66.2k
1 vote

Why are sklearn's cross_val_score values not increasing with the size of the training set?

To add to @sycorax' answer: If I understand the description of your data correctly, you have features: resistivity, density, ... (how many such physical properties do you have?) And in terms of ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible