88 votes
Accepted

CNN architectures for regression?

First of all a general suggestion: do a literature search before you start making experiments on a topic you're not familiar with. You'll save yourself a lot of time. In this case, looking at ...
user avatar
  • 15.9k
75 votes
Accepted

Should I use a categorical cross-entropy or binary cross-entropy loss for binary predictions?

Bernoulli$^*$ cross-entropy loss is a special case of categorical cross-entropy loss for $m=2$. $$ \begin{align} \mathcal{L}(\theta) &= -\frac{1}{n}\sum_{i=1}^n\sum_{j=1}^m y_{ij}\log(p_{ij}) \\ &...
user avatar
  • 78.2k
63 votes
Accepted

How is softmax_cross_entropy_with_logits different from softmax_cross_entropy_with_logits_v2?

You have every reason to be confused, because in supervised learning one doesn't need to backpropagate to labels. They are considered fixed ground truth and only the weights need to be adjusted to ...
user avatar
  • 3,164
51 votes
Accepted

Step-by-step example of reverse-mode automatic differentiation

Let's say we have expression $z = x_1x_2 + \sin(x_1)$ and want to find derivatives $\frac{dz}{dx_1}$ and $\frac{dz}{dx_2}$. Reverse-mode AD splits this task into 2 parts, namely, forward and reverse ...
user avatar
  • 9,420
50 votes

Building an autoencoder in Tensorflow to surpass PCA

Here is the key figure from the 2006 Science paper by Hinton and Salakhutdinov: It shows dimensionality reduction of the MNIST dataset ($28\times 28$ black and white images of single digits) from the ...
user avatar
  • 94.5k
48 votes

What's the difference between variance scaling initializer and xavier initializer?

Historical perspective Xavier initialization, originally proposed by Xavier Glorot and Yoshua Bengio in "Understanding the difficulty of training deep feedforward neural networks", is the weights ...
user avatar
  • 3,164
44 votes
Accepted

Adam optimizer with exponential decay

Empirically speaking: definitely try it out, you may find some very useful training heuristics, in which case, please do share! Usually people use some kind of decay, for Adam it seems uncommon. Is ...
user avatar
  • 6,732
41 votes

Should I use a categorical cross-entropy or binary cross-entropy loss for binary predictions?

There are three kinds of classification tasks: Binary classification: two exclusive classes Multi-class classification: more than two exclusive classes Multi-label classification: just non-...
user avatar
31 votes
Accepted

how to weight KLD loss vs reconstruction loss in variational auto-encoder

For anyone stumbling on this post also looking for an answer, this twitter thread has added a lot of very useful insight. Namely: beta-VAE: Learning Basic Visual Concepts with a Constrained ...
user avatar
  • 889
29 votes
Accepted

Keras: why does loss decrease while val_loss increase?

(this may be a duplicate) It looks like your model is over fitting, that is just memorizing the training data. In general a model that over fits can be improved by adding more dropout, or training and ...
user avatar
  • 506
27 votes
Accepted

Loss function for autoencoders

I think the best answer to this is that the cross-entropy loss function is just not well-suited to this particular task. In taking this approach, you are essentially saying the true MNIST data is ...
user avatar
  • 882
26 votes
Accepted

Understanding LSTM units vs. cells

The terminology is unfortunately inconsistent. num_units in TensorFlow is the number of hidden states, i.e. the dimension of $h_t$ in the equations you gave. Also, ...
user avatar
25 votes
Accepted

How does minibatch gradient descent update the weights for each example in a batch?

Gradient descent doesn't quite work the way you suggested but a similar problem can occur. We don't calculate the average loss from the batch, we calculate the average gradients of the loss function. ...
user avatar
  • 3,679
24 votes

How does one interpret histograms given by TensorFlow in TensorBoard?

Currently the name "histogram" is a misnomer. You can find evidence of that in the README. The meaning of the histogram interface might change some day as they said there. However, this is what it ...
user avatar
24 votes
Accepted

Difference between samples, time steps and features in neural network

I found this just below the [samples, time_steps, features] you are concerned with. X = numpy.reshape(dataX, (len(dataX), seq_length, 1)) Samples - This is the ...
user avatar
24 votes

Is it possible to give variable sized images as input to a convolutional neural network?

There are a number of ways to do it. Most of these have already been covered in a number of posts over StackOverflow, Quora and other content websites. To summarize, most of the techniques listed can ...
user avatar
  • 758
23 votes
Accepted

Loss function autoencoder vs variational-autoencoder or MSE-loss vs binary-cross-entropy-loss

I don't believe there's some kind of deep, meaningful rationale at play here - it's a showcase example running on MNIST, it's pretty error-tolerant. Optimizing for MSE means your generated output ...
user avatar
  • 1,924
22 votes

Is it common practice to minimize the mean loss over the batches instead of the sum?

As mentioned by pkubik, usually there's a regularization term for the parameters that doesn't depend on the input, for instance in tensorflow it's like ...
user avatar
  • 14k
22 votes

Should I use a categorical cross-entropy or binary cross-entropy loss for binary predictions?

Binary cross-entropy is for multi-label classifications, whereas categorical cross entropy is for multi-class classification where each example belongs to a single class.
user avatar
22 votes
Accepted

Difference in log base for cross entropy calcuation

log base e and log base 2 are only a constant factor off from each other: $$\frac{\log_e{n}}{\log_2{n}} = \frac{\log_e{2}}{\log_e e} = \log_e 2$$ Therefore using one over the other scales the ...
user avatar
  • 22.9k
20 votes
Accepted

Epoch Vs Iteration in CNN training

One iteration means one batch processed. One epoch means all data processed one times. So one epoch is counted when ...
user avatar
18 votes
Accepted

Deep learning : How do I know which variables are important?

What you describe is indeed one standard way of quantifying the importance of neural-net inputs. Note that in order for this to work, however, the input variables must be normalized in some way. ...
user avatar
18 votes

How is Spatial Dropout in 2D implemented?

This response is a bit late, but I needed to address this myself and thought it might help. Looking at the paper, it seems that in Spatial Dropout, we randomly set entire feature maps (also known as ...
user avatar
  • 882
18 votes

Is it possible to give variable sized images as input to a convolutional neural network?

The convolutional layers and pooling layers themselves are independent of the input dimensions. However, the output of the convolutional layers will have different spatial sizes for differently sized ...
user avatar
  • 882
17 votes
Accepted

How does Tensorflow `tf.train.Optimizer` compute gradients?

It's not numerical differentiation, it's automatic differentiation. This is one of the main reasons for tensorflow's existence: by specifying operations in a tensorflow graph (with operations on ...
user avatar
  • 22.2k
17 votes
Accepted

How to use pre trained word2vec model?

In Python, you can use Gensim ...
user avatar
  • 298
17 votes

how to weight KLD loss vs reconstruction loss in variational auto-encoder

I would like to add one more paper relating to this issue (I cannot comment due to my low reputation at the moment). In subsection 3.1 of the paper, the authors specified that they failed to train a ...
user avatar
  • 382
17 votes

Can I enforce monotonically increasing neural net outputs (min, mean, max)?

Two techniques: penalty and variable transformation. penalty build one model with these three outputs, then modify/customize the loss function during its estimation by adding the penalty for violation ...
user avatar
  • 56.4k
15 votes

Tensorflow Cross Entropy for Regression?

No, it doesn't make sense to use TensorFlow functions like tf.nn.sigmoid_cross_entropy_with_logits for a regression task. In TensorFlow, “cross-entropy” is ...
user avatar
  • 78.2k
14 votes

Relu vs Sigmoid vs Softmax as hidden layer neurons

In addition to @Bhagyesh_Vikani: Relu behaves close to a linear unit Relu is like a switch for linearity. If you don't need it, you "switch" it off. If you need it, you "switch" it on. Thus, we get ...
user avatar
  • 6,800

Only top scored, non community-wiki answers of a minimum length are eligible