I have a simple classification problem.
Input feature's range: 0 ~ 65535
Target output: 2 labels, [1, 0] or [0, 1]
Here is what my MLP looks like.
def multilayer_perceptron(x): mu = 0 sigma = 1 dim = 23 left = 30 right = 10 # Network Parameters n_hidden_1 = 128 # 1st layer number of features n_hidden_2 = 128 # 2nd layer number of features n_hidden_3 = 128 # 2nd layer number of features n_input = dim*(left+1+right) n_classes = 2 layer_1_w = tf.Variable(tf.truncated_normal([n_input, n_hidden_1], mean = mu, stddev = sigma), name='layer_1_w') layer_1_b = tf.Variable(tf.truncated_normal([n_hidden_1]), name='layer_1_b') layer_1 = tf.add(tf.matmul(x, layer_1_w), layer_1_b) layer_1 = tf.nn.relu(layer_1) layer_2_w = tf.Variable(tf.truncated_normal([n_hidden_1, n_hidden_2], mean = mu, stddev = sigma), name='layer_2_w') layer_2_b = tf.Variable(tf.truncated_normal([n_hidden_2]), name='layer_2_b') layer_2 = tf.add(tf.matmul(layer_1, layer_2_w), layer_2_b) layer_2 = tf.nn.relu(layer_2) layer_3_w = tf.Variable(tf.truncated_normal([n_hidden_2, n_hidden_3], mean = mu, stddev = sigma), name='layer_3_w') layer_3_b = tf.Variable(tf.truncated_normal([n_hidden_3]), name='layer_3_b') layer_3 = tf.add(tf.matmul(layer_2, layer_3_w), layer_3_b) layer_3 = tf.nn.relu(layer_3) out_layer_w = tf.Variable(tf.truncated_normal([n_hidden_3, n_classes], mean = mu, stddev = sigma), name='out_layer_w') out_layer_b = tf.Variable(tf.truncated_normal([n_classes]), name='out_layer_b') out_layer = tf.matmul(layer_3, out_layer_w) + out_layer_b return out_layer
After training process, I use training data(value range from 0~65535) to test my network. But the value of output layer is too large(more than 1000) or too small(less than -1000). Basically, I can have the correct answer by picking the larger value from two of them of output layer. But at the same time, I also want to put the output layter to softmax().
Here are my questions:
If I feed softmax() with the very large or very small value, I got [1, 0] or [0, 1] which is not intuitive. I want the output of softmax be something like [0.333, 0.667]. How can I achieve this by controlling the value of the output layer to be very close.
I've tried to normalize input feature to 0 ~ 1. It helps but the value of output layer is still range from -100 to 100 which still is too large to softmax().
I believe the cause of this phenomenon is deu to ReLU activation function and adding effect of many neurons. Therefore, I change the ReLU to Sigmoid. The result is what I expected, the value of output layer is closer. But I don't think it's the right way to solve this problem since changing activation funciton alter my network architecture and affect the training process.
Any opinion and experience is welcome. Thanks in advance.