2
$\begingroup$

Language: R

Background

data = 1800 observations (rows) x 5 variables (columns)

I am using library(caret) and training regression models using nnet() and would like to know whether my interpretation and choice of model is valid.

My response variable is in hours (therefore continuous and non-negative)

Method

In this example I am training three models, the only difference between them is the number of max iterations (maxit) (to try and 'smooth' the values at the high-end of the range).

I have also split my data into a training df_train and a test df_test dataset using createDataPartition().

# use 10-fold cross-validation
cvCtrl <- trainControl(method="repeatedcv", repeats=3)
modFit_1 <- train(v_response ~., method="nnet", trControl=cvCtrl, data=df_train, trace=TRUE, maxit=1000, linout = 1)
modFit_2 <- train(v_response ~., method="nnet", trControl=cvCtrl, data=df_train, trace=TRUE, maxit=2000, linout = 1)
modFit_3 <- train(v_response ~., method="nnet", trControl=cvCtrl, data=df_train, trace=TRUE, maxit=4000, linout = 1)

The plots of predicted vs actual values are for each model, and their RMSE values are:

modFit_1

RMSE = 17.31634

modFit_2

RMSE = 16.73134

modFit_3

RMSE = 9.526294

Questions

  1. As my response variable has to be non-negative would I immediately assume modFit_2 to be the 'best' model out of the three (even though modFit_3 has a lower RMSE)?

  2. Is there a way to ensure the predicted values are non-negative (and is my modFit_2 non-negative just by chance)?

Update

Distribution of v_response

v_response_distribution

Residual vs fitted plot

df_diag <- data.frame(residuals = modFit_3$finalModel$residuals, 
                        fitted = modFit_3$finalModel$fitted.values)

ggplot(data=df_diag, aes(x=fitted, y=residuals)) +
  geom_point()

residual_v_fitted

$\endgroup$
  • $\begingroup$ Very interesting plots: how did you produce the ones which show predicted vs actual values? $\endgroup$ – statisticianwannabe Jan 14 '18 at 15:32
3
$\begingroup$

Your plots of modFit_3 looks perfect to me: a straight 45° degree line and the errors look homoscedastic. Please, check the homoscedasticity by plotting errors vs predicted values. If it is so, can't you just use max(0, predicted) for a (very) few negative values?

A general way to ensure non-negativity of the response is log-transform it before training, that is

train(log(v_response) ~., method="nnet", ...)

and then inverse transform the predicted values. But this may also have other effects, in particular change error distribution. So be careful. For deciding about log-transform it would also be helpful to see the histogram of v_response

In your code you are using linout = 1, which is linear activation function. This is very similar (though not equivalent) to linear regression. It allows negative values. You can try other activation functions, such as rectifier, but this is out of the "nnet" scope. I suspect that the default value of linout (logistic) that introduces nonlinearities in the model (actually, the main purpose of neural network) did not give you satisfactory results. But if your problem is linear by nature (seems so based on your perfect modFit_3 plot), why do you bother with nnet and don't try linear regression?

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ I've updated my question with a plot of fitted v residuals, and also the distribution of v_response. (I initially tried linear regression but wasn't getting good results, so I started exploring other techniques such as trees and neural nets) $\endgroup$ – tospig Apr 23 '15 at 21:34
  • 1
    $\begingroup$ And for linout = FALSE I do indeed get incorrect results (just 1s) - as per this similar question $\endgroup$ – tospig Apr 23 '15 at 22:41
  • 1
    $\begingroup$ @tosplg Your v_response distribution has a long tail and residuals are larger for large fitted values (which means, some heteroscedasticity is there). Therefore, yes, log-transforming v_response looks reasonable in your case. $\endgroup$ – lanenok Apr 24 '15 at 9:10
  • 1
    $\begingroup$ I thought you might say that and tried it with a log(v_resposne), but it gave me a worse fitting model with very distinct patterns in the residual vs fitted plot. $\endgroup$ – tospig Apr 24 '15 at 10:57
  • 1
    $\begingroup$ @tosplg Yes, one cannot say in advance what is better. Anyway with linear activation function it is only by chance that you do not have negative values as in modFit_2 $\endgroup$ – lanenok Apr 24 '15 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.