Questions tagged [caret]
Caret is an R package containing a set of functions that attempt to streamline the process of creating predictive models.
459
questions
2
votes
1answer
21 views
Why do the results of LASSO regression differ after removing uninformative variables in glmnet?
I am researching therapy response of melanoma patients based on a number of approximately 80 features with a very small sample size of 60 patients. To eliminate features that do not contribute to the ...
0
votes
0answers
19 views
Classification problem. I have issues with overfitting/leakage, i.e. sensitivity in training is 1.0. How do I fix this?
I have a consumer dataset (size: 10200x180), I am using this to develop a training model to predict consumers who would provide a referral. I have done the following to set the model.
Cleaning:
I have ...
1
vote
1answer
34 views
Unreasonable bias when using nnet (R package caret) for time series forecasting
I have been trying to forecast a time series in a regression-like setting using neural networks (nnet method in R package caret)....
0
votes
1answer
12 views
ML test accuracy higher than training? Small unbalanced samples were stratified by class
My background is in ecology, it is common to have smaller sample sizes and class imbalances and ML approaches are still increasingly adopted. My specific dataset: training set is 49 sample, my test ...
0
votes
1answer
22 views
Impute missing values of dummy variables, using R's {caret} package: predicted values in between {0;1}?
I'm using {caret} to impute missing data resulting from non-response to survey questions. All of these variables are defined as numeric, though most are dummies. ...
0
votes
0answers
12 views
How to perform Multiple Imputation (MI) for longitudinal survey data, using {caret} in R?
I have a large dataset comprising survey responses throughout a time period of three to four decades (or in other words, slightly above 20 waves). Keep in mind that these are longitudinal data and not ...
1
vote
0answers
38 views
Extreme collinearity for Random Forest?
My data have 450 observations and 2200 predictors that I want to use to train a RF model to classify 4 classes. However, about 2165 of the predictors are very highly correlated to each other. The way ...
0
votes
1answer
45 views
Variable Importance for Caret Random Forest Regression
I have trouble understanding the exact meaning of the feature importance scores in caret for RF regression. As you know there are many potential importance measures for RF. However, there is no clear ...
0
votes
0answers
44 views
0
votes
0answers
13 views
Machine learning methods for filtering noisy data
I have my data (hear beat) cleaned up with the appropriate filter, however, they are still noisy and need to extract a clean pattern. Which is the best algorithm for this purpose?
I have also clean ...
0
votes
1answer
80 views
Model building with 10-fold validation
I have yet to find a sufficient and succinct answer regarding model building with 10-fold cross validation (in this case, using Caret). I've found responses here, for instance: https://stackoverflow....
0
votes
0answers
44 views
How to manually calculate predictions of kernlabs SVM
I am trying to manually replicate the predictions of kernlabs SVM (polynomial & radial kernel) using caret.
Here is the code to fit the model:
...
1
vote
1answer
46 views
How many models does caret::trainControl() actually train?
I searched around CV and the caret documentation but couldn't find an answer to this. I'm using caret in R to tune some random forests for classification trained ...
1
vote
1answer
55 views
0
votes
1answer
145 views
Cross-validation using Caret in R: Why are coefficients from FinalModel identical to those from lm()?
I think I must be missing some fundamental part of the logic of cross-validation, or machine learning in general.
Using the caret package in R, I ran a repeated k-...
2
votes
0answers
47 views
Which R-Squared formula is used in carets' k-fold-cross validation function
I am currently trying to implement a linear model and calculate some out-of-sample accuracy metrics while using k-fold-cross validation. I try to do this using R's caret package, so this question ...
0
votes
0answers
41 views
Interpret varimp() with Caret LDA model
I'm using caret's train() function for a binary classification outcome with an lda models, based on several features
...
1
vote
0answers
41 views
Drop unused parameters from model?
I have trained a model using caret's train function in R (LASSO logistic regression with glmnet, repeatedcv). The goal is both feature selection and getting the model. With the preferred lambda, LASSO ...
1
vote
0answers
22 views
KNN and Classification Factor
I'm studying the KNN algorithm and its application on R. If I correctly understood, KNN is a supervised algorithm able to classify an unlabeled item according to the predominant belonging class in the ...
1
vote
1answer
133 views
What really is 'glmnet' when used in caret in R for binary classification?
like lasso and ridge, elastic net can also be used for classification
by using the deviance instead of the residual sum of squares. This
essentially happens automatically in caret if the response ...
7
votes
2answers
194 views
How should two cross-validated logistic regression models be compared?
I'm using 100 times 10-fold repeated cross-validation to assess the ROC-AUC performance improvement of adding a biomarker to an existing model:
Model_A : pred1 + pred2
Model_B :pred1 + pred2 + pred3
I'...
1
vote
0answers
100 views
2
votes
1answer
51 views
Model stacking, Super Learner Algorithm
I've recently started studying ensembles in ML, particularly Super Learner Algorithm. To be honest, although I have read several articles related to this topic, I am a little bit confused. I want to ...
0
votes
1answer
76 views
Confusion Matrix for Multi-class Variables using cross-validation
my dataset includes multi-class variables (11 different variables). All of the columns are numeric except the last column which is the label ( Last Column Name = Movement, 11 different types, Type = ...
0
votes
0answers
79 views
Permutation Testing Machine Learning Model Performance With Caret in R
Using the caret package in R, classification machine learning model performance is usually assessed using the confusionmatrix() function, which uses a Binomial Test ...
0
votes
0answers
12 views
What can go wrong if we don't scale and center data in a predictive model? [duplicate]
I have started using R recently, along with the caret package. I note that there is an option to preprocess data so that it is scaled and centered.
I am interested in multiple linear regression. What ...
0
votes
0answers
65 views
Random Forest doesn't converge in caret
I'm using some synthetic data to try and understand the performance of different ML algorithms on my real data. I'm finding that RF is consistently overfitting the training data even though its ...
1
vote
1answer
45 views
What's random in Caret if the hold out is fixed for cross validation [closed]
I tried to fix the folds for cross validation in Caret. But there is still some randomness returned by the train function. See code below. Don't they should return the same?
...
2
votes
1answer
52 views
RFE: Pre-define a specificity threshold
I would like to use recursive feature elimination (implemented via caret in R) to perform feature selection for about 40 test results with 2 possible outcomes. Consequently, RFE either models by ...
1
vote
1answer
249 views
How do I calculate confidence intervals on an elastic net regression in R
I am performing an elastic net regression on my data n = 34, p = 46
I first built the model using the "caret" package with the cross validation method to set the optimal alpha and lambda ...
2
votes
0answers
125 views
Elastic net: how to tune for both lambda and alpha? Why is caret better than cv.glmnet?
I'm learning how to use R to fit Elastic Net models using the glmnet package. There appears to be a function, cv.glmnet that ...
0
votes
0answers
27 views
Why is Rsquared smaller in whole sample than k-fold cross-validated?
I want to get a cross-validation $R^2$ for a linear regression model with one predictor to get a measure of how well my model predicts rather than explains data (in R with package caret). I assumed ...
0
votes
1answer
75 views
What does the featurePlot() function from the caret package actually do?
Just a quick example from Max Kuhn's book.
...
0
votes
0answers
30 views
Can I use caret RFE result for subsequent random forest with CV?
I've been googling and reading alot about my issue but couldn't find a clear answer.
In order to prevent data leakage, I use caret RFE for feature elimination:
...
1
vote
0answers
27 views
How to apply svm to a dataset with a numerical (not categorical) dependent variable?
Context: I have some band values from a sentinel - 2 derived .tiff
I now want to make a prediction regarding areas where i have no actual field data
i.e. make a carbon map of sorts
Libraries are:
<...
1
vote
0answers
50 views
is a final model different from cross validation?
It appears that I have completely misunderstood cross validation for several months so I want your help clarifying the idea using an example rather than only theory from some great SE questions. My ...
1
vote
2answers
53 views
How does repeated cross validation go about 'averaging' model coefficients?
No matter how much I google, I cannot find the answer to this simple question.
Say you do 10-fold, repeated (5x) CV logistic regression with elastic net regularization.
For alpha you try ...
1
vote
0answers
12 views
Weighted observations for linear model [closed]
I have dataset with sale price of products in years 2015-2020. I would like use it to predict future sale prices. I would like to use sale year as a weight of observation (eg. for 2020-100%, for 2019-...
1
vote
1answer
34 views
How can I implement 5 times repeated 10 fold cross validation using the randomForest package ( instead of caret)?
I would like to eventually use the PIMP-Algorithm (Permutation Variable Importance Measure) in order to get p values for the variables' importance. However, the formula
...
0
votes
1answer
25 views
Difference Between Built-In Cross Validation Functions and Using Caret
I was wondering if someone could provide some insight on the pros/cons of using built-in cross validation functions like cv.glmnet (https://www.rdocumentation.org/packages/glmnet/versions/3.0-2/topics/...
0
votes
0answers
65 views
How to group factor levels for stepwise regression using caret
Using the train() function from caret in R, I'm trying to run a stepwise ANCOVA, but each level of my 9-level factor is being ...
2
votes
0answers
137 views
How to calculate variable importance for different models? is varImp() the solution?
I'm using caret's train() function for a binary classification outcome with different models (nb, knn, lda, qda, glm, rpart, rf).
I'm using varImp() and plots to determine the importance of every ...
1
vote
1answer
18 views
Importance value (with varImp from carret package) for one of the two numerical predictors has value 100, how do I interpret this?
I'm using two numerical predictors to find an outcome, when using varImp (from the carret package) one of the predictors has 100 importance and the other 0.
How should I interpret this?
1
vote
1answer
59 views
Why can't my ROC reach a low PPV?
I am creating a couple of models (RF, SVM, LR) and I want to evaluate all of them on a certain PPV (0.7).
This question and this question helped me write my code:
...
0
votes
0answers
47 views
Multi-classification: low precision due to imbalanced classes in test data - what to do?
I built a multi-classification model with 3 result classes (XGBoost using R's caret-package): A, B and C. I undersampled my training data - so every class is equally abundant for training. The ...
0
votes
0answers
24 views
Kappa for More than Two Classes in R?
I am building predictive models in R using the oil dataset which has 7 classes. I am wanting to use Kappa for optimization since the classes are extremely ...
0
votes
0answers
12 views
Caret for model performing and tuning paramter
I would like to ask, if I could use cross validation in caret for tuning parametr lambda and also for evaluating model performance. If I use savePredictions="final" and also I will tune parametr, it ...
1
vote
1answer
186 views
Help testing the predictive quality of a binomial GLM (currently attempting using the “caret” package)
Hello world (sorry for the novel; if you read this, I appreciate it!),
I'm running into a question that is probably a mixture of how to approach a problem of modeling and the technical difficulties ...
1
vote
1answer
706 views
LOOCV in Caret package ( randomForest example) - not unique results
I pose you my doubts:
For what I know there is only a single way to perform a LOOCV for a model (i.e. testing each one of the N elements vs the model trained with the other N-1 elements).
Namely, ...
0
votes
1answer
38 views
Caret GBM predict only produces 106 outputs while newdata has 403 rows… What's up?
I am training ML classifiers using caret to predict mortality in a clinical data set. Training with caret's gbm works well, but when I try to use predict, I get very strange results. Here is my code:
...
