2
$\begingroup$

Architecture:

I have a CNN which does some classification for me. The output layer y consists of a vector $\vec{y}$ which is of dimension $(1, 1000)$, so it has 1.000 neurons in total (the weight matrix $W_{out}^{5}$ between the last fully-connected layer (layer 5) and the output layer is of dimension $(500, 1000)$, since layer 5 has 500 neurons).

Instead of a one-hot encoding my teacher signal is made of a gaussian distribution. So, e.g., when the real value would be class 500 (so the center of vector $\vec{y_{real}}$) a gaussian distribution with a mean/location of 500 is fit into $\vec{y_{real}}$ and fed to the network as the teacher signal. Plotted the teacher signal looks like this then: gaussian distribution for a real value with class 500.

This works quite well and makes sense, since all 1000 classes are related to each other - their information share one "abstract category". Additionally I want to get information about the noise in the current input...so I interpret the shape and variance of the distribution in my output layer ($\vec{y}$) as information about my process' noise.

To give another example: if the real value is 500 I am totally happy if the network's output $\vec{y}$ looks somehow like a gaussian distribution with a global maximum in range from like 490 to 510.

Scenario: Two different outputs?

However, my network contains information about a second "abstract category" (which has nothing in common with the first category). This leads to my current problem: I want to have the network predict both categories, each classified by the network via outputting (optimally) a gaussian distribution.

What would be an appropriate solution for this scenario?

I thought of altering my output layer to be of dimension $(2, 1000)$ first...but I am not sure if 1000 different classes are appropriate for my second category and additionally I do not know if it makes any sense to have the last fully-connected layer be connected to an output of dimension $(2, 1000)$, especially because the two categories (and therefore the two distributions I want to have) have nothing in common semantically.

My second idea was to have two different output vectors, $\vec{y_1}$ for the first category and $\vec{y_2}$ for the second category...but how would the cost function look like then? I guess it would not make any sense to calculate two different errors for category 1 and 2 and then learning the network with those errors in each epoch?

Any ideas on this topic?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Sure, this is entirely possible and you already suggested the answer: you have two output neurons, both fully connected to the previos layer in the network. How the cost function would look like actually depends on the cost function you use. If it is the squared loss, then the cost function is just the loss of class 1 plus the loss of class 2. Backpropagation also doesn't change too much in this case: you simply add up the derivatives of both "sub-loss" functions.

If good predictions of one output class are harder to achieve than the other, then you should balance your loss function (e.g. 30% for the easy one, 70% for the harder one). These weights then get absorbed into the derivates as well.

Improve this answer
$\endgroup$
16
  • $\begingroup$ You got me wrong (probably should have made this clearer in the main post): by $\vec{y}$ having a dimension of $(1, 1000)$ I mean $\vec{y}$ is already a vector with 1.000 entries, therefore I have 1.000 neurons predicting the 1.000 classes of the first category. So $\vec{y_{real}}$ is a gaussian distribution of 1.000 discrete values with a certain mean/location fitted into the training signal/vector. Now, I want to have two of those outputs predicting between a vast amount of classes with normal distributions. $\endgroup$
    – daniel451
    Commented Feb 7, 2016 at 15:22
  • $\begingroup$ Ok, let me get this straight.You essentially have two target vectors for your network, on both you have applied gaussian smoothing to sidestep the problems that come with a 1-of-n representation. The CNN should predict 2000 classes now? In this case the answer is still the same, you just output 2000 neurons and the entire network shares its weights to predict both. But I guess this is still not what you are after. What specifically do you mean by "classified by a Gaussian". You classify using the CNN, the Gaussian smoothing is a side effect. Could you be more specific, perhaps share some data? $\endgroup$
    – pAt84
    Commented Feb 7, 2016 at 16:30
  • $\begingroup$ You are right, this is not what I want. What I mean is the following: I got one abstract category, $cat_1$. This is classified by the network (1.000 output neurons) via outputting (optimally) a gaussian distribution. The teacher signal is a gaussian distribution with its mean/location at the real value/class. Now I have another category $cat_2$ which has nothing in common with $cat_1$. I believe my network has information about both categories, so I want my network to output one vector (1.000 neurons, gaussian) to predict the class of $cat_1$ and another vector to predict class of $cat_2$. $\endgroup$
    – daniel451
    Commented Feb 7, 2016 at 16:42
  • $\begingroup$ To clarify my goal: instead of having two different networks, each with an output vector of like 1.000 neurons, to predict class of $cat_1$ respectively $cat_2$, I want one network to do this - because the input data is the same. $\endgroup$
    – daniel451
    Commented Feb 7, 2016 at 16:47
  • $\begingroup$ By teacher signal you mean target variables, yes? Essentially a number of data vectors where each vector is 1000D and has a a gaussian form when you just line up the output neurons (like in the image above). Now the big question is, does this second "abstract class" also have a teacher signal / target variables or do you believe it is somehow embedded in the CNN and should "magically" create another 1000 outputs? $\endgroup$
    – pAt84
    Commented Feb 7, 2016 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.