2
$\begingroup$

I'm using Caret to perform some analyses, but even after reading some books and the caret manual I still have some theoretical doubts. Below the code I'm using.

#-- Partition of to original data into test and training
#-- A two column df, response and predictor.
part <- createDataPartition(lmMatrix$IC50 , p = 0.70, list = F)
training <- lmMatrix[part,]
testing <- lmMatrix[-part,]

#-- CV method and model training 
tc <- trainControl(method = "LOOCV", number = 1, repeats = 1)
fit <- train(IC50 ~ ES, data = lmMatrix, trControl = tc, method = "lm", metric = "RMSE")

#-- Prediction with test data
pred <- predict(fit, newdata = testing, interval = "confidence")

#-- Get RMSE Testing set - prediction of testing test
RMSE <- c(RMSE, mean((testing$IC50 - pred)^2))

The questions that I have are:

  • Shouldn't I repeat this several times to get an average of the predicted output of the test data?

  • In this case I just have a response and a predictor, when doing the training, the final model is just a regular linear model fit or is it tuned (which is the reason for CV).

  • Finally, in the case of having more than one predictor B1 and B2, the tuning made in the train function means that the final model may be one of these: y = B1 + B2, y = B1 x B2, y = B1, y = B2 Is this the real meaning of tuning of the model or I'm totally misunderstanding this.

$\endgroup$
2
  • $\begingroup$ Can you edit to reframe this as the approach you are using, rather than the code? Can you add comments & pseudocode to illustrate your approach for people who don't use R? (Asking for code help, &/or code check, is generally off topic here.) $\endgroup$ Nov 7 '16 at 12:19
  • $\begingroup$ @gung excellent suggestion!. but from the question described by OP, i think may be OP might not able to have an abstracted description and asking for specific software implementation. $\endgroup$
    – Haitao Du
    Nov 7 '16 at 12:21
3
$\begingroup$

Shouldn't I repeat this several times to get an average of the predicted output of the test data?

The caret library will automatically do it for you and select the best "hyper-parameters" for the model (if the model has parameters to tune)

In this case I just have a response and a predictor, when doing the training, the final model is just a regular linear model fit or is it tuned (which is the reason for CV).

In fact, for lm and glm there you are using cross validation incorrectly. Because there is no "hyper-parameter" to tune, and just coefficient in the model. The difference between "hyper-parameter" and coefficient is that hyper parameter gives the "general template" of the model, and the "coefficient" gives details. Think about polynomial regression, where the "hyper-parameter" gives the order of the polynomial, and the "coefficient" gives coefficient for each term. Cross validation are used for "model selection" on "polynomial orders", but not "model training" on learning coefficients.

Finally, in the case of having more than one predictor B1 and B2, the tuning made in the train function means that the final model may be one of these: y = B1 + B2, y = B1 x B2, y = B1, y = B2 Is this the real meaning of tuning of the model or I'm totally misunderstanding this.

No. For caret, lm has no "hyper-parameters", it will just give you the performance on different "re-sampled" data set.


To experience "model selection" and "parameter" learning, I would recommend you try "ridge regression" in caret (using method=glmnet),

$$ \min (\|X\beta-y\|^2+\lambda\|\beta\|^2) $$

where $\lambda$ is the hyper-parameter. And once you select it, learning to coefficients $\beta$ is similar to fit a linear model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.