10
This is uncommon and can be quite risky, especially when there are multiple local minima for the cost function. You might even end up in the neighbourhood of a local maximum after averaging them out!
This is similar (not the same since subsets are not bootstrap samples) to ensembling (e.g. random forests). Not very common, and not preferable when training ...
3
When you run cv.glm, the data are split into parts, with a model fitted to one part and predictions calculated on the other part. The fit and predict steps both call cut and the cutpoints will typically be different. That means the levels of the factor cut(age,12) will be different in the fitting data and the prediction data. R is warning you that the levels ...
2
No. The main difference is that CV attempts to emulate out of sample performance of the model while GoF tests are in sample. Roughly speaking GoF looks at the fit to entire sample while CV fits to a subset then compares the fit with the other part of dataset
2
Feature selection is not always necessary. I approach it with a number of questions/decisions:
Does feature selection make sense for the type of data at hand, i.e. can we expect from our knowledge about the application and data generation processes that there are substantial numbers of features that do not carry information?
In my case, I often work with ...
answered Jun 11 at 21:40
cbeleites unhappy with SX
30k2 gold badges61 silver badges120 bronze badges
2
This was clarified in comment
So isn't cross validation technically a technique for preventing
overfitting, which is also the purpose of regularization?
Not exactly. For example, imagine that your data has $n$ samples and $p$ features with $n \ll p$. In such case fitting a multivariate regression model to all the features would overfit, so you need a model ...
2
The typical procedure is to choose the best hyperparameters, which OLS doesn't have, based on cv performance and train the model on the entire training set (as you expected to be). The procedure that the student employs is actually choosing a dataset. And, it increases the variance because if any of the folds have a bad partition, you may end up choosing ...
2
IMHO your best bet is repeated k-fold cross validaton or out-of-bootstrap (possibly also .632+-bootstrap, depending on the actual risk of overfitting).
These resampling methods test in turn all cases, and that's the best in terms of CI you can get with such a small data set.
They also test an arbitrarily large number of surrogate models, allowing you to ...
answered Jun 24 at 22:40
cbeleites unhappy with SX
30k2 gold badges61 silver badges120 bronze badges
1
Yes, the typical cross validation assumes iid samples, so that it can freely split the data into training and validation. In case of dependency, such as the temporal dependency in time series datasets, modifications respecting this dependency should be done for the splitting of data. Otherwise, there will be data leakage. See the following for an example of ...
1
The source code of cv.glm shows the call places for the cost function.
The other calls to cost are used to calculate the bias corrected MSE.
In the docs for cv.glm it mentions as:
delta
A vector of length two. The first component is the raw
cross-validation estimate of prediction error. The second component is
the adjusted cross-validation estimate. The ...
1
You are in a situation where the number of potential predictors (3000) far exceeds the number of cases (60), so you can't just do ordinary linear regression. With only 60 cases you probably can only include 4 to 6 effective predictors in your final model to avoid overfitting (10-15 cases per predictor). You have to find a principled way to cut down the ...
1
Calculation of MSE cross-validation is typical, but calculating $\theta_{cv}$ as yours is not good in general. Recently, a similar question has been asked; see option (1) in the question. If $X^TX$ is not singular, the problem has only one minimum, and based on your data, you might not see any adverse effects of averaging out $\theta_k$. But still, I wouldn'...
1
What you are describing is K-fold cross-validation. It's defined in terms of the number of times you train and evaluate the model (k) instead of the amount of data you leave for validation, but they are functionally equivalent. 10-fold cross-validation is equivalent to your leave-10%-out cross-validation.
This answer describes an implementation using caret.
1
There are multiple issues present.
The first one is that otm.arxiv() does not follow the standard R practice of returning a fitted model that one applies forecast() to. Instead, it performs fitting and forecasting. To obtain a forecast from otm.arxiv(), you need to supply an h parameter to it:
otm.arxiv(y,h=3,thetaList=seq(from=1,to=5,by=0.5),g="SE&...
1
As Tim mentioned in the comment, regularization and cross validation are different things. And they often are used together.
For example, we know we have the overfitting problem, and we want to regularize the model. But how much to regularize? We need cross validation to tell us, i.e., the we split the data set into two sets, and tune the regularization ...
1
Your implementation is ok. I cannot see on the webpage you referred to, where the difference in mtry comes from. I will not be surprised if you end up with slightly different mtry if you train on full data or just on a training part of the data like you did. Most likely if you check the difference in accuracy / ROC or whichever metric used, the difference is ...
1
As was already pointed out by the answer of cebeleites, inner and outer CV loop have different purposes: the inner CV loop is used to get the best model, the outer CV loop can serve different purposes. It can help you to estimate in a more unbiased way the generalisation error of your top performing model. Additionally it gives you insights into the "...
Only top voted, non community-wiki answers of a minimum length are eligible
