New answers tagged cross-validation
0
Refer to a recent answer to a similar question, in particular the last half of it. This also inspired an vignette example along the same lines.
However, you need to have your estimates on the log scale first. For example, fit the model with log(metabolite) as the response.
0
I dont know answer and cant comment,
but if I can give my simple approach: think you are using your model to help others as you offering service based on your model.
(like start planting seeds when temp raise above 0 for next 5 months)
my guess is training/test splits of data represent customer only little:
*customer is more likely to dont know if model was ...
1
They are equivalent for the exact reason that you mention: one is a monotonic transformation of the other. Just keep in mind that we want to maximize $R^2$, while we want to minimize the residual sum of squares.
Since we typically want to find the best model for a set of data, we typically have a constant $TSS$.
0
I've never quite seen NxN, but I think it may be referring to this scenario:
If N examples are available, N training sets (subsamples) with N-1 examples are created, such that each of those N training sets (subsamples) omits exactly one (qty. 1) example that (collectively) every other training set (subsamples) did not omit.
This strategy is a limiting case ...
1
I agree with the accepted answer.
Just a comment on your code:
In your trainControl function you specify
method="repeatedcv", number=k.folds, repeats=3.
Repeatedcv will automatically create 3 different partitions of 7 folds.
However, you also specify index=df1.folds
As far as I understandd, this conflicts with the above since it provides your ...
1
... I use train and test sets for building and testing my model (this
includes all the preprocessing steps and nested cv) and use the valid
set to test my final model.
Test set is typically used as final evaluation and validation set for tuning. So, I'll be using the general convention below.
Do we use Nested Cross validation to tune the hyperparameters ...
0
To me it seems to be a data problem. You are splitting the data 70:30, but are all data from the 70% prior to data from the 30% set?
It can be a problem if you mix older and newer data in the training set. If time is involved in the generation of data, which seems to be the case as you have live data, test set should never contain data that were generated ...
0
I second Stephan's answer that the likely culprit is overfitting the entire dataset. That said, another thing to validate is that there are no differences between data processing pipelines in your training vs. production code. E.g. are you normalizing the features before training? If so, do you record the means and standard deviations and apply the same ...
2
I believe you are looking for Stratified K-Fold Cross Validation, which preserves the distribution of classes across folds of the data.
Scikit-learn has a StratifiedKFold function for Python users!
11
It's hard to say without digging deeply into your model and your data. However, it seems like you have been doing a lot of cross-validation, model tuning, cross-validation, model tuning and so forth. That, together with bad out-of-sample performance, suggests that you are overfitting to your test set. That is harder than overfitting in-sample (which is easy ...
0
Thanks for the responses so far! That has helped me a lot. If I take a nested CV for the training and then evaluate it on a hold-out dataset, the AUC ROC still fluctuates massively!
I have shown this once here:
[,1]
[1,] 0.8489011
[2,] 0.8401598
[3,] 0.7405095
[4,] 0.8031968
[5,] 0.7604895
[6,] 0.8653846
[7,] 0.8231768
[8,] 0.8551449
[9,] ...
1
When the test performance relies too much on the random split, it's good practice to do nested cross-validation for test set performance. But, with this method, you won't end up with a champion model but an estimate of real data performance when you apply your training strategy.
The overall performance, e.g. RMSE or AUC, is always calculated on the test set ...
0
Hint: Start with the dual SVM objective with $\ell_2$ regularization
$$\text{maximize}\quad J(\mathbf{\Lambda}) = \sum_{i=1}^n \lambda_i - \frac{1}{2} \sum_{i=1}^n \sum_{j=1}^n \lambda_i \lambda_j y_i y_j \mathbf{K}(\mathbf{x}_i, \mathbf{x}_j)$$
$$\text{subject to}\quad \lambda_i \geq 0; \; \forall \; i.$$
Note that we take the hard-margin SVM, because of ...
0
So, this is a fundamental limitation in ML (the covariate shift problem) -- any distribution shift over your covariates is going to cause you to lose performance in expectation -- so without any knowledge of what the shift looks like, you're doomed.
Your problem seems to be related to domain adaptation, where given a small amount of unlabeled data from a new ...
0
The code you show seems to fit some kind of generalized linear model (see the method="glm" bit, perhaps a logistic regression with a logit-link function) to the data and picks hyper-parameters / evaluates based on 10-fold cross-validation. It's not clear to me what metric is being optimized (per binary log-loss, perhaps accuracy, perhaps AUC).
In ...
1
Validation Set is only necessary when we have hyperparameters in our model, otherwise validation is useless.
You are right in that when no hyperparameters are tuned a single split into training and testing is all you usually do for an internal* generalization error estimate.
Validation is however, a somewhat ambiguous term here (see here for my take on the ...
answered Nov 17 at 10:54
cbeleites unhappy with SX
33.7k3 gold badges66 silver badges131 bronze badges
1
When doing LOOCV, is there a difference between mean absolute error, and root mean square error? As you're only testing on a single data point, I think they should be the same?
No, they aren't. After all, you are not testing on a single data point, but on each separate data point, and then aggregating the average or squared errors by averaging.
As an ...
4
One thing that is not widely appreciated is that over-fitting the model selection criteria (e.g. validation set performance) can result in a model that over-fits the training data or it can result in a model that underfits the training data.
This example is from my paper (with Mrs Marsupial)
Gavin C. Cawley, Nicola L. C. Talbot, "On Over-fitting in ...
0
You can certainly do inner k-fold CV for model optimization and a single (outer) 15% split for estimating generalization error.
However, as Frank Harrel commented, this is inefficient:
when doing a single split of your available data, you encounter exactly the same risks for not splitting into independent subsets that you encounter e.g. with k-fold. So no ...
answered Nov 9 at 17:37
cbeleites unhappy with SX
33.7k3 gold badges66 silver badges131 bronze badges
1
Here's the trick:
Each case is tested in some (exactly one) fold in each run (iteration, repetition) of the cross validation.
After b runs, we have trained a total of bk surrogate models. Out of which, b were used to test/predict a (one) given case.
Each of those b predictions comes from a different surrogate model, but since we're looking at only one case (...
answered Nov 9 at 16:05
cbeleites unhappy with SX
33.7k3 gold badges66 silver badges131 bronze badges
-1
As discussed here data splitting is in general a bad idea because of arbitrariness and wasted sample size, and it requires a huge sample size to work. More details are available in the model validation chapter of RMS.
2
Everything you do to the training set needs to be done without the test set, and validation set data. So the proper way to do this is to estimate the mean and SD for normalization of your data on the training set and then use these estimates to also normalize the test set and the validation set. It might not always make a huge difference, nevertheless, this ...
10
Consider a population $Y|X$ that follows some distribution according to a true model, and you have a set of trained models $f(X,\theta)$ that make predictions of $Y$ given $X$ and are parameterized by $\theta$.
The goal is to find out what the error of the models is, in making predictions about samples from the population, as function of the parameter $\...
1
The meaning of 'model stability' is open to interpretation. An application of control theoretic stability for statistical models are not well established, at least there is no single definition out there. However, the repeated k-fold CV can be used for nonparametric way of finding generalisation performance as an interval estimate, as single CV would be a ...
2
The question is very long, and covering it in full would require a very lengthy answer. So here I will try to provide my view with very brief bullet points.
The question over-emphasises the difference between machine learning and statistics. For a good reflection on those I recommend reading the answer by Michael I. Jordan which was given during his Q&A ...
5
Is it true to some extent that statisticians are usually more
concerned about the model's goodness-of-fit and the corresponding
metrics of significance, and not that much about model's
generalization capability, and vice versa for the ML scientists?
No. Measuring generalisation capability is a large portion of statistics practice. Cross validation and ...
7
The answer is no, too many claims in your post, not only to the main question.
The difference between ML and stats is arbitrary, superficial, and not important. There is plenty of statisticians that are doing predictive models where the main goal is, of course prediction. Common machine learning methods have been developed by statisticians and published in ...
11
Is it true to some extent that statisticians are usually more concerned about the model's goodness-of-fit and the corresponding metrics of significance, and not that much about model's generalization capability, and vice versa for the ML scientists?
Scientists and analysts using "pure" statistics have recently gotten into some trouble precisely ...
0
The typical rule of thumb is to use modes (principal components, "PCs") if their associated eigenvalue is greater than one. However, it might be better to use an eigenvalue threshold based on the Marcenko-Pastur law:
\begin{equation}
\lambda^+=\left(1+ \sqrt{\frac{p}{n}} \right )^2
\end{equation}
where $p$ is the number of variables and $n$ is the ...
2
The process of building a model requires one to understand a large number of issues. Go back to fundamentals and study some texts and articles and case studies first. See for example RMS. You'll see that model specification is something that needs serious thought and subject matter knowledge, and that it is not wise to use the data to tell you which model ...
Top 50 recent answers are included