Language: R
Background
data = 1800 observations (rows) x 5 variables (columns)
I am using library(caret)
and training regression models using nnet()
and would like to know whether my interpretation and choice of model is valid.
My response variable is in hours (therefore continuous and non-negative)
Method
In this example I am training three models, the only difference between them is the number of max iterations (maxit
) (to try and 'smooth' the values at the high-end of the range).
I have also split my data into a training df_train
and a test df_test
dataset using createDataPartition()
.
# use 10-fold cross-validation
cvCtrl <- trainControl(method="repeatedcv", repeats=3)
modFit_1 <- train(v_response ~., method="nnet", trControl=cvCtrl, data=df_train, trace=TRUE, maxit=1000, linout = 1)
modFit_2 <- train(v_response ~., method="nnet", trControl=cvCtrl, data=df_train, trace=TRUE, maxit=2000, linout = 1)
modFit_3 <- train(v_response ~., method="nnet", trControl=cvCtrl, data=df_train, trace=TRUE, maxit=4000, linout = 1)
The plots of predicted vs actual values are for each model, and their RMSE
values are:
RMSE = 17.31634
RMSE = 16.73134
RMSE = 9.526294
Questions
As my response variable has to be non-negative would I immediately assume
modFit_2
to be the 'best' model out of the three (even thoughmodFit_3
has a lower RMSE)?Is there a way to ensure the predicted values are non-negative (and is my
modFit_2
non-negative just by chance)?
Update
Distribution of v_response
Residual vs fitted plot
df_diag <- data.frame(residuals = modFit_3$finalModel$residuals,
fitted = modFit_3$finalModel$fitted.values)
ggplot(data=df_diag, aes(x=fitted, y=residuals)) +
geom_point()