Is it possible to train neural networks such that certain activations are rewarded and some other activations are penalized? In other words, I would like the network to generate preferred values more often. An examples of this would be:

Assume that a neuron has 25 inputs and the values of inputs are restricted to [-1, 1] interval. I have quantized the inputs so that each input has 3 possible values. As a result, there are 3^25 potential combinations of the inputs. I would like the majority of these combinations to produce an output that is greater than 0.5. Additionally, the smaller portion of combinations produce values in the [-1, 0.5) range.

Can I do this by somehow modifying the activation function? If yes, how should I enforce the requirements?

In general, would a network that is trained this way be able to perform classification as well as a network that does not have such limitations?


1 Answer 1


I'm not sure what do you mean by 'preferring' some activations, but there are ways to 'encourage' something similar.

  1. Prefer activations that are bigger than a certain threshold, e.g, positive activations

If you use ReLU (Rectified Linear Unit) as activation function, you'll only get positive values.

  1. Prefer activations that are within a certain range, e.g., [-0.5, 0.5]

You can use activation function that clips activations to your range.

It would be also helpful if you shared something on context, since there may be other techniques that work in a similar way (for example Batch Normalization sort of encourages activations to be standardized, and weight decay also has some effect on activations because it penalizes 'big' weights).

  • $\begingroup$ You're right. I think the examples I provided didn't reflect what I had in mind well. I updated my post with a better example. $\endgroup$
    – Matt
    Commented Feb 19, 2018 at 1:07
  • $\begingroup$ In my previous examples, for example in 1., I do not want to eliminate the chance of having negative values, I just want to see positive values more often than negative values. $\endgroup$
    – Matt
    Commented Feb 19, 2018 at 1:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.