Find Neural Network Inputs Given Outputs

Question

I've trained a neural network with two inputs, a single hidden layer with two neurons, and one output using a bipolar sigmoid activation function. If a single input is known, how would I determine the second input to create a desired output?

For example, let's say the neural network is trained to add two inputs to produce an output. So if input_1 = 3 and input_2 = 4, the output will be 7, (3 + 4 = 7). Given input_1 = 3 and the desired output is 7, I want to calculate the second input required to produce the desired output (the answer should be 4).

How would I do this for a network that is more complicated than basic addition and has multiple inputs/outputs? For example, for a network with four inputs and two outputs, how would I calculate input_3 and input_4 given input_1, input_2, output_1 and output_2?

Answer 1

I'm no expert in this field, so I might be wrong. Therefore, correct me if I'm wrong.

consider this neural network (which I suppose is equivalent to yours):

A---H1
 \ /  \
  X    C
 / \  /
B---H2

consider that the activation function of H1, H2 and C is the bipolar sigmoid, to which we'll refer to as "bsig(x)"

also, we'll name the connections as follows:
A, H1: wa1;
A, H2: wa2;
B, H1: wb1;
B, H2: wb2;
H1, C: wh1;
H2, C: wh2

now the values of H1, H2 and C can be defined as:

H1 = bsig(wa1 * A + wb1 * B)
H2 = bsig(wa2 * A + wb2 * B)
C = bsig(wh1 * H1 + wh2 * H2)

So, C can be written as:

C = bsig(wh1 * bsig(wa1 * A + wb1 * B) + wh2 * bsig(wa2 * A + wb2 * B))

All you need to do is solve this equation in order to B or A depending on which of the values is unkown.

Answer 2

The answer is simple: backpropagation.

Say you have a trained net $f$ which maps some $x$ to some $y$, which you'd ideally want to be $z$. You then use a different loss, which is the deviation from $y$ to $z$, e.g.:

$$ \mathcal{C} = ||z - y||_2^2. $$

In back propagation, you typically follow the gradient of the loss $\mathcal{L}$ with respect to the weights via stochastic gradient descent. Say you have a weight matrix $W$, then you do: $$ W \leftarrow W - \eta {\partial \mathcal{L} \over \partial W} $$ where $\eta$ is some learning rate. Now, to obtain the right $x$ to get out $z$, you do the same thing with the inputs on $\mathcal{C}$ for several iterations: $$ x \leftarrow x - \eta {\partial \mathcal{C} \over \partial x}, $$ where you have initialised $x$ randomly. Since this optimisation will typically be non convex, you might want to start this with different initialisations.

Practically, you will have to compute the derivatives for the input layer (e.g. the $\delta$ in most text books).

Answer 3

You could use reconstruction error and consider your second input as a parameter. Suppose your fixed input is $x_1$ and your latent input is $x_2$. The output $y$ can be:

$$y = s(W_1 \cdot x_1 + W_2 \cdot x_2 + b)$$

$s$ being your bipolar sigmoid.

You then use a decoder to map back into a reconstruction $z$:

$$z = s(W' \cdot y + b')$$

Then the error function to minimize could be the squared error:

$$L(xz) = || x - z ||^2$$

You can now train your network the way you want, with $W_1, W_2, b, W', b'$ AND $x_2$ being parameters of your gradient.

Answer 4

I agree with the backpropagation. However, if the question is more general, especially for bigger nets, you can train another network with many generated input outputs from your original network and approximate the backpropagation with the 2nd network.

Hope it helps

Answer 5

I believe bernardo may have made an error. At the hidden layer h there my not be a function sigmoid since sigmoid is the function applied to h1 and h2 to give result c. Therefore to correct his assumption h1 = awa1 + bwb1. Hence, h2 follows suit. Rendering the function sigmoid to process h1 and h2. C= sig(h1 + h2) Then solve for input. This may work and per linear algebra we know there maybe a number of inputs that can result in target c. The only caveat to this who solution is to assume the weight matrix has already been trained. If it has then it is simply trained for input a and b. In other words the only way i can think this can be valuable is to determine if other inputs can result in the same solution.