2
$\begingroup$

So say I want the user to type the first 5 letters of a word he or she is thinking about; I want the neural network to output the remaining letters in the word.

However, should I:

  • Convert the 5-letter string to binary (A=01000001, B=01000010, etc.)
  • This means that I will have 5 * 8 = 40 input neurons

Or should I:

  • Link letters to numbers (A=1, B=2, etc.)
  • Divide this number by ALPHABET_LENGTH=26 (normalize)
  • Resulting in A~0.04, B~0.08, etc.
  • Meaning I will only have 5x1 input neurons

The same would count for the output of course!

However, what is more effective? Which normalization will have a more positive effect on the neural-network training (through backpropagation and genetic algorithms, both). And are there any papers on this?

$\endgroup$
5
  • $\begingroup$ I am not sure what you mean by more effective in this case. $\endgroup$ Commented Apr 7, 2017 at 15:15
  • $\begingroup$ @MichaelChernick edited $\endgroup$ Commented Apr 7, 2017 at 15:17
  • $\begingroup$ It strikes me as highly implausible (even irrelevant) that the ASCII encoding of letters (as proposed in the first instance) would have anything useful to do with your objective. $\endgroup$
    – whuber
    Commented Apr 7, 2017 at 15:19
  • $\begingroup$ @whuber don't see it as 'encoding'; it's just a way of normalization. Each bit represents an input neuron in that case. Secondly, why would you think it's useless? $\endgroup$ Commented Apr 7, 2017 at 15:22
  • $\begingroup$ It's both: it starts out as an encoding and then gets normalized. That doesn't change the fact that it's a coding method. It's irrelevant because the bits used in ASCII likely have nothing to do with predicting work completions. That implies you have perfect freedom to choose other (completely arbitrary) encodings or, possibly, encodings that might have some intrinsic meaning (and thereby assist the NN to make good predictions). $\endgroup$
    – whuber
    Commented Apr 7, 2017 at 15:25

1 Answer 1

2
$\begingroup$

Instead of guessing the best encoding for the prediction, let the neural network do it for you!

The way to do this, especially if you have a limited range of possible input characters, is start with one-hot encoding. I.e. A = 0000...0001, B = 0000...0010, etc.

Next, let the first layer in your neural network be an embedding layer - this will project the one-hot encoding into a more dense vector representation that contains information about the different relationships between the characters.

The next layers of your neural network will start with this dense vector representation and go from there.

To understand better how this embedding process works, have a look at word2vec. This illustrates the process of creating embeddings for words in large texts. You can find more information here for example: https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/

Adding an embedding layer is particularly easy when using keras.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.