Hot answers tagged

37

Lets say your caret model is called "model". You can access the final glmnet model with model$finalModel. You can then call coef(model$finalModel), etc. You will have to select a value of lambda for which you want coefficients, such as coef(model$finalModel, model$bestTune$.lambda). Take a look at the summaryFunction parameter for the trainControl ...


33

There must be some features in your training set with class 'char' . Please check this > a <- c("1", "2",letters[1:5], "3") > as.numeric(a) [1] 1 2 NA NA NA NA NA 3 Warning message: NAs introduced by coercion


32

To the train function in caret, you can pass the parameter na.action = na.pass, and no preprocessing (do not specify preProcess, leave it as its default value NULL). This will pass the NA values unmodified directly to the prediction function (this will cause prediction functions that do not support missing values to fail, for those you would need to specify ...


29

According to the caret manual, page 22, the parameter repeats only applies when the method is set to repeatedcv, so no repetition is performed when the method is set to cv. So the difference between both methods is indeed that repeatedcv repeats and cv does not. Aside: Repeating a crossvalidation with exactly the same splitting will yield exactly the same ...


21

Don't preProcess the data prior to running the train function! Use the preProcess argument for the train function, and the pre-processing will be applied to each re-sampling iteration. e.g. don't do this: library(caret) dat <- iris pp <- preProcess(dat[,-5], method="pca") dat[,-5] <- predict(pp, dat[,-5]) knnFit1 <- train(Species~., dat, ...


20

Ok, here is my try: boot - bootstrap boot632 -- 0.632 bootstrap cv -- cross-validation, probably this refers to K-fold cross-validation. LOOCV -- leave-one-out cross validation, also known as jacknife. LGOCV -- leave-group-out cross validation, variant of LOOCV for hierarchical data. repeatedcv -- is probably repeated random sub-sampling validation, i.e ...


19

It looks like Max Kuhn actually started working on a package for ensembleling caret models, but hasn't had time to finish it yet. This is exactly what I was looking for. I hope the project gets finished one day! edit: I wrote my own package to do this: caretEnsemble


17

I didn't see the lecture, so I can't comment on what was said. My $0.02: If you want to get good estimates of performance using resampling, you should really do all of the operations during resampling instead of prior. This is really true of feature selection [1] as well as non-trivial operations like PCA. If it adds uncertainty to the results, include it ...


17

What is the issue with #1? It runs fine for me and the result of the call to varImp() produces the following, ordered most to least important: > varImp(modelFit) rpart variable importance Overall V5 100.000 V4 38.390 V3 38.362 V2 5.581 V1 0.000 EDIT Based on Question clarification: I am sure there are better ways, but here is how I might do ...


16

You can specify method="none" in trainControl. For example: train(Species ~ ., data=iris, method="rf", tuneGrid=data.frame(mtry=3), trControl=trainControl(method="none")) I'm not sure when this was implemented.


16

In theory, the performance of a RF model should be a monotonic function of ntree that plateaus beyond a certain point once you have 'enough' trees. This makes ntree more of a performance parameter than a Goldilocks parameter that you would want to tune. Caret tends to focus on tuning parameters that perform poorly for high and low values in which you want to ...


16

What you have displayed is a classic example of overfitting. The small uptick in error comes from poorer performance on the validation portion of your cross-validated data set. More iterations should nearly always improve the error on the training set, but the opposite is true for the validation/test set.


15

One way to think about the process of building a predictive model (such as a neural network) is that you have a 'budget' of information to spend, much like a certain amount of money for a monthly household budget. With only 87 observations in your training set (and only 36 more in your test set), you have a very skimpy budget. In addition, there is much ...


14

train does tune over both. Basically, you only need alpha when training and can get predictions across different values of lambda using predict.glmnet. Maybe a value of lambda = "all" or something else would be more informative. Max


14

The difference is the pre-processing. predict.train automatically centers and scales the new data (since you asked for that) while predict.randomForest takes whatever it is given. Since the tree splits are based on the processed values, the predictions will be off. Max


14

In general, boosting error can increase with the number of iterations, specifically when the data is noisy (e.g. mislabeled cases). This could be your issue, but I wouldn't be able to say without knowing more about your data Basically, boosting can 'focus' on correctly predicting cases that contain misinformation, and in the process, deteriorate the ...


14

I see two issue here. First, your training set is too small relative to your testing set. Normally, we would want a training set that is at least comparable in size to the testing set. Another note is that for Cross Validation, you're not using the testing set at all, because the algorithm basically creates testing sets for you using the "training set". So ...


12

So using 10-fold CV will split the data into 10 different sets of roughly the same size. The model is fit on 90% and the remaining 10% is used to estimate accuracy. This process continues "round robin" 9 more times. The accuracy is the average of the 10 holdouts for each tuning value. For example: > set.seed(1) > train_control <- trainControl(...


11

There is a pretty easy way, assuming tune <- train(...): probsTest <- predict(tune, test, type = "prob") threshold <- 0.5 pred <- factor( ifelse(probsTest[, "yes"] > threshold, "yes", "no") ) pred <- relevel(pred, "yes") # you may or may not need this; I did confusionMatrix(pred, test$response) Obviously, you can set threshold ...


11

I haven't gotten around to implementing it for all the models that can accept weights. Right now, it should work for rpart variants, glmnet, gamSpline, glmboost, gamboost, evtree, ctree, ctree2, chaid, cforest, blackboost, treebag, glm, glmStepAIC, and bayesglm. Note that ksvm function does not have a weight parameter, so those models won't be enabled.


11

Probably the cause is you have some character variables in your data frame. Convert all character variable into factor in one line: library(dplyr) data_fac=data_char %>% mutate_if(is.character, as.factor)


10

Old question, but I recently had to deal with this problem and found this question as a reference. Here is an alternative approach: The glmnet vignette (https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html) specifically addresses this issue, recommending to specify the cross validation folds using the foldids argument and validate $\lambda$ over a ...


10

By default, caret keeps the components that explain 95% of the variance. But you can change it by using the thresh parameter. # Example preProcess(training, method = "pca", thresh = 0.8) You can also set a particular number of components by setting the pcaComp parameter. # Example preProcess(training, method = "pca", pcaComp = 7) If you use both ...


9

One thing you might want to look into are regularized random forests, which are specifically designed for feature selection. This paper explains the concept, and how they differ from normal random forests Feature Selection via Regularized Trees There's also a CRAN package RRF that's build on the randomForest that will allow you to implement them easily in ...


9

The best way would be to explicitly supply the tuneGrid dataframe. For instance, random forest has only one tuning parameter, 'mtry', which controls the number of features selected for each tree. To set mtry at a specific value, you might choose the randomForest default (?randomForest) do this: model <- train(x = X, y = Y, method = 'rf', tuneGrid = data....


9

I suspect your y is of class numeric and is not an R factor. You can look at the documentation for glmnet directly, y: response variable. Quantitative for ‘family="gaussian"’ or ‘family="poisson"’ (non-negative counts). For ‘family="binomial"’ should be either a **factor with two levels, or a two-column matrix of counts or proportions**...


9

As I understood you have only 3 variables. By default varImp function returns scaled results in range 0-100. Var3 has the lowest importance value and its scaled importance is zero. Try to call varImp(rf, scale = FALSE).


9

To address both your questions. The discrepancy between rfFit and rfFit$finalModel I believe it is normal to have some discrepancy between your rfFit and rfFit$finalModel. As you can see in the output from rfFit there is also a Accuracy SD column. Your Accuracy returned here is an average as a result of your repeated cross validation. The Accuracy ...


8

What you are looking for is called "model ensembling". A simple introductory tutorial with R code can be found here: http://viksalgorithms.blogspot.jp/2012/01/intro-to-ensemble-learning-in-r.html


8

Caret does let you tune the number of trees on its backend randomForest package. For instance, considering the latest version (4.6-12) as of now, you just pass the normal ntree parameter. caret will "repass" it to randomForest, e.g.: train(formula, data = mydata, method = "rf", ntree = 5, trControl = myTrControl)


Only top voted, non community-wiki answers of a minimum length are eligible