How to encourage certain activations during training of neural networks?

Question

Is it possible to train neural networks such that certain activations are rewarded and some other activations are penalized? In other words, I would like the network to generate preferred values more often. An examples of this would be:

Assume that a neuron has 25 inputs and the values of inputs are restricted to [-1, 1] interval. I have quantized the inputs so that each input has 3 possible values. As a result, there are 3^25 potential combinations of the inputs. I would like the majority of these combinations to produce an output that is greater than 0.5. Additionally, the smaller portion of combinations produce values in the [-1, 0.5) range.

Can I do this by somehow modifying the activation function? If yes, how should I enforce the requirements?

In general, would a network that is trained this way be able to perform classification as well as a network that does not have such limitations?

Answer 1

I'm not sure what do you mean by 'preferring' some activations, but there are ways to 'encourage' something similar.

1. Prefer activations that are bigger than a certain threshold, e.g, positive activations

If you use ReLU (Rectified Linear Unit) as activation function, you'll only get positive values.

2. Prefer activations that are within a certain range, e.g., [-0.5, 0.5]

You can use activation function that clips activations to your range.

It would be also helpful if you shared something on context, since there may be other techniques that work in a similar way (for example Batch Normalization sort of encourages activations to be standardized, and weight decay also has some effect on activations because it penalizes 'big' weights).