I have a situation where I'd like a DNN to learn the [unknown] mapping between two fixed-length strings. A [simplistic] example:

"-+--++-++-" -> "968"
"+-+-+-+-+-" -> "185"
"-+-+-+++--" -> "766"

I can normalize the characters in the input string to convert them into the numerical inputs required by the DNN but I'm not sure how to structure the output layer (I'm using Keras).

Assuming the output string is 3 characters long:

model = models.Sequential()
model.add(layers.Dense(x1, activation='relu', input_shape=(N,)))
model.add(layers.Dense(x2, activation='relu'))
model.add(layers.Dense(3, activation='???'))
  1. I'll need a way to convert the 3 outputs back into characters.
  2. I can't seem to figure out which activation function I should use.
  3. I'll also need to choose appropriate optimizer and loss functions


model.compile(optimizer='???', loss='???')
  • $\begingroup$ I think it would help if you described what the strings mean. $\endgroup$ – Aaron Aug 25 '17 at 17:02

I actually suggest making the output layer Dense(n*k) where $n$ is the number of symbols in the ouput string and $k$ is the alphabet size. Then reshape the output to $n$ by $k$ and apply softmax along the $k$ axis. This will result in a matrix $M$ where $M_{ij}$ is the probability that the $i$th letter is symbol $j$.

At training time, you shouldn't bother with converting the output into characters. Instead, convert the target string to a 1-hot encoding, which will also be an $n$ by $k$ matrix. Apply standard categorical crossentropy loss between the 1-hot encoded target and matrix $M$.

As for optimizer, it usually doesn't matter much. Either Adam or RMSProp should be able to get you good results. For activation, I suggest relu, but the last layer before the softmax should have no activation.

At test time, in order to retrieve the actual output characters, the $i$th output character will be $\text{argmax}_j M_{ij}$.

  • $\begingroup$ Thank you so much. Do you agree that simply normalizing the input string (into an array of floats) makes sense for the first/input layer? $\endgroup$ – RobertJoseph Aug 26 '17 at 12:04
  • 1
    $\begingroup$ Yes, if the string is binary, then encoding as 0.0's and 1.0's should work fine. If it is not binary, then try using a 1-hot encoding for each symbol. $\endgroup$ – shimao Aug 26 '17 at 16:54
  • $\begingroup$ Here is a quick code example of how I've tried to implement your suggestions. I would appreciate it if you'd take a quick look. $\endgroup$ – RobertJoseph Aug 29 '17 at 18:49
  • 1
    $\begingroup$ That does not look completely correct. To be honest, i'm not familiar enough with keras to tell you what the correct way to do this is. However, in tensorflow, it would look something like: out = tf.nn.softmax(x, dim = 1) where both out and x are shaped n by k. Also, another complication is that we are not dealing with a categorical distribution anymore, but an array of categorical distributions (since each predicted symbol is a categorical variable!), therefore using keras' built in categorical cross-entropy loss may not work. You might have to write your custom loss metric. $\endgroup$ – shimao Aug 30 '17 at 2:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.