2
$\begingroup$

So say I want the user to type the first 5 letters of a word he or she is thinking about; I want the neural network to output the remaining letters in the word.

However, should I:

  • Convert the 5-letter string to binary (A=01000001, B=01000010, etc.)
  • This means that I will have 5 * 8 = 40 input neurons

Or should I:

  • Link letters to numbers (A=1, B=2, etc.)
  • Divide this number by ALPHABET_LENGTH=26 (normalize)
  • Resulting in A~0.04, B~0.08, etc.
  • Meaning I will only have 5x1 input neurons

The same would count for the output of course!

However, what is more effective? Which normalization will have a more positive effect on the neural-network training (through backpropagation and genetic algorithms, both). And are there any papers on this?

Improve this question
$\endgroup$
5
  • $\begingroup$ I am not sure what you mean by more effective in this case. $\endgroup$
    – Michael R. Chernick
    Apr 7, 2017 at 15:15
  • $\begingroup$ @MichaelChernick edited $\endgroup$
    – Thomas Wagenaar
    Apr 7, 2017 at 15:17
  • $\begingroup$ It strikes me as highly implausible (even irrelevant) that the ASCII encoding of letters (as proposed in the first instance) would have anything useful to do with your objective. $\endgroup$
    – whuber ♦
    Apr 7, 2017 at 15:19
  • $\begingroup$ @whuber don't see it as 'encoding'; it's just a way of normalization. Each bit represents an input neuron in that case. Secondly, why would you think it's useless? $\endgroup$
    – Thomas Wagenaar
    Apr 7, 2017 at 15:22
  • $\begingroup$ It's both: it starts out as an encoding and then gets normalized. That doesn't change the fact that it's a coding method. It's irrelevant because the bits used in ASCII likely have nothing to do with predicting work completions. That implies you have perfect freedom to choose other (completely arbitrary) encodings or, possibly, encodings that might have some intrinsic meaning (and thereby assist the NN to make good predictions). $\endgroup$
    – whuber ♦
    Apr 7, 2017 at 15:25

1 Answer 1

Reset to default
2
$\begingroup$

Instead of guessing the best encoding for the prediction, let the neural network do it for you!

The way to do this, especially if you have a limited range of possible input characters, is start with one-hot encoding. I.e. A = 0000...0001, B = 0000...0010, etc.

Next, let the first layer in your neural network be an embedding layer - this will project the one-hot encoding into a more dense vector representation that contains information about the different relationships between the characters.

The next layers of your neural network will start with this dense vector representation and go from there.

To understand better how this embedding process works, have a look at word2vec. This illustrates the process of creating embeddings for words in large texts. You can find more information here for example: https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/

Adding an embedding layer is particularly easy when using keras.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.