Unexpected Negative Partial Derivatives for Input Features to Neural Network Using a relatively standard MLP neural network, I model the total duration of an activity based on many counts of sub-activities, where the relationships between the sub-activities and duration may be non-linear. I mean-center and z-normalize the inputs for insertion into the network, meaning that some input examples have negative values for some features.
I want to understand what sub-activities have the greatest impact on total activity duration, across the input space. I use sensitivity analysis to do this. 
Here, for each input example, I calculate the partial derivative of the response with respect to each feature. However, I get a mixture of positive and negative derivatives!
A negative derivative for an input means that the response (total duration) decreases when we increase the input (number of sub activities). Of course, this makes no sense, because a sub activity always takes a small non-negative amount of time to occur.
Does anyone have any idea why I would sometimes be getting these negative derivatives?
 A: Neural network weights are trained by using some optimization method. Optimization doesn't care about physical impossibility unless you tell it to. It sounds like in your problem you'd like to enforce a particular monotonic constraint on the network, because doing any activity takes some positive amount of time. There's some literature about how to enforce monotonicity on neural networks.
Here's an example of an early publication in this vein.
Joseph Sill. "Monotonic Networks". California Institute of Technology. 1998.

Monotonicity is a constraint which arises in many application domains. We present a machine learning model, the monotonic network, for which monotonicity can be enforced exactly, i.e., by virtue of functional form. A straightforward method for implementing and training a monotonic network is described. Monotonic networks are proven to be universal approximators of continuous, differentiable monotonic functions. We apply monotonic networks to a real-world task in corporate bond rating prediction and compare them to other approaches.

