I've been experimenting with various simple neural networks to test their performance. When I use the following architecture, I'm getting some very bad test error, which looks like overfitting.

$$\text{My architecture}: \mathbb{R}^n \ni \texttt{Input}\xrightarrow{\text{affine}} \mathbb{R}^m \xrightarrow{\text{dropout}} \mathbb{R}^m \xrightarrow{\text{Max(0, x) (ReLU)}} \mathbb{R}^m \xrightarrow{\text{affine}} \texttt{Output} \in \mathbb{R}$$

enter image description here

This seems to persist over various values of $n$ and $m$, and over various rates of dropout. (Although with lower probabilities of keeping nodes, it seems to be worse. So the more likely dropout is to happen, the more this seems to occur. The above picture is for dropout probability $.8$ during training.)

I'm planning on doing some more experiments to investigate just what is happening and how it's overfitting. But before I do, I want to make sure I'm not making some stupid mistake in my implementation.

Question: Is there some obvious mistake in my implementation that's making this network perform so poorly?

The guts of my network look like this:

def dropout_network(self, weights, biases, x, noise_on):
    current_dropout_rate = tf.cond(
                    lambda: np.float32(1), 
                    lambda: self.dropout_rate
    fc1 = tf.add(tf.matmul(x, weights['wl1']) , biases['bl1'])       
    fc1 = tf.nn.dropout(fc1, keep_prob=current_dropout_rate)
    fc1 = tf.nn.relu(fc1)
    out = tf.add(tf.matmul(fc1, weights['wl2']) , biases['bl2'])
    return out

The current_dropout_rate business is to allow me to stop the dropout at test time, by feeding in noise_on: False.

One thing that makes me suspicious is that if I replace the dropout layer with other things like batch normalization, or just the identity, nothing like this happens. That's not necessarily suspicious, since of course dropout is very different from these, and I'm using dropout in a very simplified setting. But dropout sure is performing badly.

For reference, the function my network is trying to fit is a randomly selected cubic function $\mathbb{R}^{10} \to \mathbb{R}$. My test data and training data are drawn from a standard normal distribution. Training data is $5000 \times 10$, and test is $1250 \times 10$.


closed as off-topic by whuber Aug 4 '18 at 17:30

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – whuber
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ don't you have the condition inverted? $\endgroup$ – shimao Aug 4 '18 at 6:53
  • $\begingroup$ @shimao Yes! Glad I asked this question. $\endgroup$ – Eric Auld Aug 4 '18 at 14:35
  • $\begingroup$ @shimao Any idea why this would cause the network to perform so poorly? Just turning on dropout during testing caused it to perform ten times worse! $\endgroup$ – Eric Auld Aug 4 '18 at 14:59

I had transposed the conditions on the tf.cond.


Not the answer you're looking for? Browse other questions tagged or ask your own question.