0
$\begingroup$

I have a simple classification problem.
Input feature's range: 0 ~ 65535
Target output: 2 labels, [1, 0] or [0, 1]
Here is what my MLP looks like.

def multilayer_perceptron(x):
    mu = 0
    sigma = 1

    dim = 23
    left = 30
    right = 10

    # Network Parameters
    n_hidden_1 = 128 # 1st layer number of features
    n_hidden_2 = 128 # 2nd layer number of features
    n_hidden_3 = 128 # 2nd layer number of features

    n_input = dim*(left+1+right)
    n_classes = 2

    layer_1_w = tf.Variable(tf.truncated_normal([n_input, n_hidden_1], mean = mu, stddev = sigma), name='layer_1_w')
    layer_1_b = tf.Variable(tf.truncated_normal([n_hidden_1]), name='layer_1_b')
    layer_1 = tf.add(tf.matmul(x, layer_1_w), layer_1_b)
    layer_1 = tf.nn.relu(layer_1)

    layer_2_w = tf.Variable(tf.truncated_normal([n_hidden_1, n_hidden_2], mean = mu, stddev = sigma), name='layer_2_w')
    layer_2_b = tf.Variable(tf.truncated_normal([n_hidden_2]), name='layer_2_b')
    layer_2 = tf.add(tf.matmul(layer_1, layer_2_w), layer_2_b)
    layer_2 = tf.nn.relu(layer_2)

    layer_3_w = tf.Variable(tf.truncated_normal([n_hidden_2, n_hidden_3], mean = mu, stddev = sigma), name='layer_3_w')
    layer_3_b = tf.Variable(tf.truncated_normal([n_hidden_3]), name='layer_3_b')
    layer_3 = tf.add(tf.matmul(layer_2, layer_3_w), layer_3_b)
    layer_3 = tf.nn.relu(layer_3)

    out_layer_w = tf.Variable(tf.truncated_normal([n_hidden_3, n_classes], mean = mu, stddev = sigma), name='out_layer_w')
    out_layer_b = tf.Variable(tf.truncated_normal([n_classes]), name='out_layer_b')
    out_layer = tf.matmul(layer_3, out_layer_w) + out_layer_b
    return out_layer

After training process, I use training data(value range from 0~65535) to test my network. But the value of output layer is too large(more than 1000) or too small(less than -1000). Basically, I can have the correct answer by picking the larger value from two of them of output layer. But at the same time, I also want to put the output layter to softmax().

Here are my questions:

  1. If I feed softmax() with the very large or very small value, I got [1, 0] or [0, 1] which is not intuitive. I want the output of softmax be something like [0.333, 0.667]. How can I achieve this by controlling the value of the output layer to be very close.

  2. I've tried to normalize input feature to 0 ~ 1. It helps but the value of output layer is still range from -100 to 100 which still is too large to softmax().

  3. I believe the cause of this phenomenon is deu to ReLU activation function and adding effect of many neurons. Therefore, I change the ReLU to Sigmoid. The result is what I expected, the value of output layer is closer. But I don't think it's the right way to solve this problem since changing activation funciton alter my network architecture and affect the training process.

Any opinion and experience is welcome. Thanks in advance.

$\endgroup$
  • $\begingroup$ 1) Why are you normalizing to 0 to 1, and not to something symmetric around 0, like -1 to 1. 2) There are initialization strategies where you let the variance in the initialization depend on the number of layers, to control the variance in the output of the first iteration, but unfortunately I have no good source for that, but maybe that's what you're looking for $\endgroup$ – Sam Aug 31 '17 at 7:55
  • $\begingroup$ @Sam 1)In my experiment, 0 to 1 or -1 to 1 makes no big difference to output layer. 2) I think softmax output layer is just a method to make things seen prettier. They can't mean the real probability of the output labels. What I mean is if a label No.1 has highest output value, I can say No.1 is the label the model predict. If the second highest label is No.2, then I can say No.2 is the second possible answer. BUT, I can't use the output value of softmax to quantize the REAL probabiblity. $\endgroup$ – Lion Lai Aug 31 '17 at 8:20
  • $\begingroup$ Why do you think that? To my best knowledge, softmax outputs are to be interpreted as class probabilistic predictions. Basically, the reason why it's bad if the weights are initialized in a way that your outputs are close to 0 or 1, is because in these areas the gradient of the softmax is close to 0, and hence backprop will be VERY slow (as the first derivative is basically 0). That's why you want to initialize in a way that the outputs are around 0.5, because there you get a large gradient. Ultimately, if your softmax converges to [0,1], that's good, it means you make confident predictions $\endgroup$ – Sam Aug 31 '17 at 9:28
  • $\begingroup$ I think you are right, but I still don't know how to control the value of output layer in a desired range. Do you know what keyword I should search on google? $\endgroup$ – Lion Lai Aug 31 '17 at 10:37
  • $\begingroup$ Depending on your level of mathematics and interest, you can either look at this paper arxiv.org/pdf/1704.08863.pdf, or search for "neural net initialization strategies" on google - or maybe something tensorflow-specific like this stackoverflow.com/questions/44284580/… ? $\endgroup$ – Sam Aug 31 '17 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.