The "standard" way to do this would be to use a CNN which learns character embeddings. The first step implicitly encodes characters in your vocabulary as one-hot vectors. In your example, this might mean encoding "a" as $[1, 0, \ldots, 0]$ and "b" as $[0, 1, 0, \ldots, 0]$. The definition of "one-hot vector" implies that the sum of two one-hot vectors is not a one-hot vector, so I'm not sure where the notion of a + b = b came from.
In the embedding layer, the network would look up the embedding corresponding to each character and convolve a set of $K$ filters across those embeddings, $H$ characters at a time. For example, if your embeddings are of dimension $D$ and each filter is of size $D \times H$, then, for a sequence of length $N$ with a stride $S$ of $1$ and no padding, the activation of each filter would be of size $N-H+1$. For $K$ filters, you would have $K$ such activations. These can then be propagated into subsequent layers (e.g. convolutional, recurrent, pooling etc.) to perform some downstream task.
If you have distinguishing features like commas in the middle of your text, max pooling should be able to capture that. For example, if you're trying to classify something, this might involve:
- performing a character-level convolution to extract meaningful features in your string, resulting in a $(N-H+1) \times K$ matrix;
- max pooling over time to determine degree of presence of each feature (such as commas) as well as to standardize the dimension to $K$;
- classification using a softmax layer with a $K\times C$ weight matrix and bias vector of size $C$
Attention could be used, for instance, in the following way for the classification problem:
- perform a character-level convolution as above
- take the inner product of a learned $K$-dimensional weight vector $\tilde{w}$ with each of the weight $N-H+1$ activations
- compute the softmax to obtain $N-H+1$ attention scores $w$
- perform a weighted average of the $N-H+1$ activations using $w$
- classify using a softmax layer with a $K\times C$ weight matrix and bias vector of size $C$
My hunch is that this would be slightly less effective since you're effectively average pooling instead of max pooling, and the averaging over $N-H+1$ activations could wash out the presence of a comma (which would be indicated by a large value in one of your $K$ dimensions) while the max pooling would pull it out if it was there in even one of them.