How to deal with discrete and continuous output multi variables in neural network?

Question

I have created neural networks using nnet for either discreate or continous output variables, but not using both at once. Now I have a problem in which the output contains both discrete and continuous variables, like this:

Y1              Y2             Y3  Y4   Y5
0.3821665898    1.2883648663    X   E   1
0.6514063927    0.5815594038    Y   B   10
0.2811264971    0.5610311362    X   D   4
0.4887534697    1.1842930657    X   A   5
0.5851027465    0.6844661487    Y   B   11
0.8273730366    1.0426096583    X   C   6
0.9663713202    1.2532121355    X   D   7
0.5929174765    1.4039754421    Y   E   5
0.8357351425    1.1997673572    Y   C   4

where Y1 and Y2 are continuous output variable and Y3-Y5 are discrete variables. Can I use discrete and continuous variable together as an output to train a neural network using nnet?

If we come across discrete variables in features we could transform them into a continuous form, like this:

X1  A   B   C
A   1   0   0
B   0   1   0
C   0   0   1
A   1   0   0
C   0   0   1
C   0   0   1

where X1 is the real feature and A,B, and C are transformed columns. Do we need to do such transformation in output variables also? What should else I looking for when doing such kind of neural network training?

Answer 1

Using a mix of continuous and categorial outputs as they are with nnet does not work:

> model <- nnet(x = iris[,1:3], y = iris[,4:5], size = 3, linout = T, maxit = 10000)

# weights:  20
Error in nnet.default(x = iris[, 1:3], y = iris[, 4:5], size = 3, linout = T,  : 
  NA/NaN/Inf in foreign function call (arg 2)
In addition: Warning message:
In nnet.default(x = iris[, 1:3], y = iris[, 4:5], size = 3, linout = T,  :
  NAs introduced by coercion

Therefore, as you did, one-hot encoding categorial features into dummy variables is the way to go. You should consider two things when using continuous and dummy variables in nnet output at the same time:

• nnet was designed to have either logical or continuous outputs (see the linout, entropy, and softmax parameters). You therefore cannot get both logical and continuous output at the same time. Using logical output with continuous data will not work by concept, so you would need to first use lineout=T, then derive the logical output you want to have from the linear output yourself.

• consider properly scaling your data, e.g. mean=0 and SD=1 (you might encounter weird effects otherwise).

Here's the small reference example I just used to check that this actually works:

# example data: feature 1-3 are IN, feature 4-5 are OUT
exampleData <- iris
# one-hot encoding
library(caret)
d <- scale(data.frame(predict(dummyVars(~., data = exampleData), exampleData)))
# fit model
model <- nnet(x = d[,1:3], y = d[,4:7], size = 3, linout = T, maxit = 10000)
# used model to predict on training data (overfitting! use test data instead!)
predicted <- predict(model, d[,1:3])
plot(predicted[,1], d[,4], main = 'Petal.Width', ylab = 'observed')
plot(predicted[,2], d[,5], main = 'Species.setosa', ylab = 'observed')
plot(predicted[,3], d[,6], main = 'Species.versicolor', ylab = 'observed')
plot(predicted[,4], d[,7], main = 'Species.virginica', ylab = 'observed')

As you see, the class output (which should be logical) is continuous instead. But the separation between classes is clearly visible, so making a class probability/class membership out of it using a logistic function should be easy.