DNN: Mapping a fixed length string to another fixed length string

I have a situation where I'd like a DNN to learn the [unknown] mapping between two fixed-length strings. A [simplistic] example:

"-+--++-++-" -> "968"
"+-+-+-+-+-" -> "185"
"-+-+-+++--" -> "766"


I can normalize the characters in the input string to convert them into the numerical inputs required by the DNN but I'm not sure how to structure the output layer (I'm using Keras).

Assuming the output string is 3 characters long:

model = models.Sequential()
...

1. I'll need a way to convert the 3 outputs back into characters.
2. I can't seem to figure out which activation function I should use.
3. I'll also need to choose appropriate optimizer and loss functions

--

model.compile(optimizer='???', loss='???')

• I think it would help if you described what the strings mean. – Aaron Aug 25 '17 at 17:02

I actually suggest making the output layer Dense(n*k) where $n$ is the number of symbols in the ouput string and $k$ is the alphabet size. Then reshape the output to $n$ by $k$ and apply softmax along the $k$ axis. This will result in a matrix $M$ where $M_{ij}$ is the probability that the $i$th letter is symbol $j$.
At training time, you shouldn't bother with converting the output into characters. Instead, convert the target string to a 1-hot encoding, which will also be an $n$ by $k$ matrix. Apply standard categorical crossentropy loss between the 1-hot encoded target and matrix $M$.
At test time, in order to retrieve the actual output characters, the $i$th output character will be $\text{argmax}_j M_{ij}$.
• That does not look completely correct. To be honest, i'm not familiar enough with keras to tell you what the correct way to do this is. However, in tensorflow, it would look something like: out = tf.nn.softmax(x, dim = 1) where both out and x are shaped n by k. Also, another complication is that we are not dealing with a categorical distribution anymore, but an array of categorical distributions (since each predicted symbol is a categorical variable!), therefore using keras' built in categorical cross-entropy loss may not work. You might have to write your custom loss metric. – shimao Aug 30 '17 at 2:06