# Neural network: two output vectors?

Architecture:

I have a CNN which does some classification for me. The output layer y consists of a vector $\vec{y}$ which is of dimension $(1, 1000)$, so it has 1.000 neurons in total (the weight matrix $W_{out}^{5}$ between the last fully-connected layer (layer 5) and the output layer is of dimension $(500, 1000)$, since layer 5 has 500 neurons).

Instead of a one-hot encoding my teacher signal is made of a gaussian distribution. So, e.g., when the real value would be class 500 (so the center of vector $\vec{y_{real}}$) a gaussian distribution with a mean/location of 500 is fit into $\vec{y_{real}}$ and fed to the network as the teacher signal. Plotted the teacher signal looks like this then: .

This works quite well and makes sense, since all 1000 classes are related to each other - their information share one "abstract category". Additionally I want to get information about the noise in the current input...so I interpret the shape and variance of the distribution in my output layer ($\vec{y}$) as information about my process' noise.

To give another example: if the real value is 500 I am totally happy if the network's output $\vec{y}$ looks somehow like a gaussian distribution with a global maximum in range from like 490 to 510.

Scenario: Two different outputs?

However, my network contains information about a second "abstract category" (which has nothing in common with the first category). This leads to my current problem: I want to have the network predict both categories, each classified by the network via outputting (optimally) a gaussian distribution.

What would be an appropriate solution for this scenario?

I thought of altering my output layer to be of dimension $(2, 1000)$ first...but I am not sure if 1000 different classes are appropriate for my second category and additionally I do not know if it makes any sense to have the last fully-connected layer be connected to an output of dimension $(2, 1000)$, especially because the two categories (and therefore the two distributions I want to have) have nothing in common semantically.

My second idea was to have two different output vectors, $\vec{y_1}$ for the first category and $\vec{y_2}$ for the second category...but how would the cost function look like then? I guess it would not make any sense to calculate two different errors for category 1 and 2 and then learning the network with those errors in each epoch?

Any ideas on this topic?

• You got me wrong (probably should have made this clearer in the main post): by $\vec{y}$ having a dimension of $(1, 1000)$ I mean $\vec{y}$ is already a vector with 1.000 entries, therefore I have 1.000 neurons predicting the 1.000 classes of the first category. So $\vec{y_{real}}$ is a gaussian distribution of 1.000 discrete values with a certain mean/location fitted into the training signal/vector. Now, I want to have two of those outputs predicting between a vast amount of classes with normal distributions. – daniel451 Feb 7 '16 at 15:22
• You are right, this is not what I want. What I mean is the following: I got one abstract category, $cat_1$. This is classified by the network (1.000 output neurons) via outputting (optimally) a gaussian distribution. The teacher signal is a gaussian distribution with its mean/location at the real value/class. Now I have another category $cat_2$ which has nothing in common with $cat_1$. I believe my network has information about both categories, so I want my network to output one vector (1.000 neurons, gaussian) to predict the class of $cat_1$ and another vector to predict class of $cat_2$. – daniel451 Feb 7 '16 at 16:42
• To clarify my goal: instead of having two different networks, each with an output vector of like 1.000 neurons, to predict class of $cat_1$ respectively $cat_2$, I want one network to do this - because the input data is the same. – daniel451 Feb 7 '16 at 16:47