# What is the Cost Function for Neural Network with Dropout Regularisation?

For some context, I shall outline my current understanding:

Considering a Neural Network, for a Binary Classification problem, the Cross-entropy cost function, J, is defined as:

$$J = \frac{-1}{m} \sum_{i=1}^m y^i*log(a^i) + (1-y^i)*log(1-a^i)$$

1. m = number of training examples
2. y = class label (0 or 1)
3. a = output prediction (value between 0 and 1)

Dropout regularisation works as follows: For a given training example, we randomly shut down some nodes in a layer according to some probability. This has the effect of keeping the weights low during training and hence regularises the network and prevents overfitting.

I have learnt that if we do apply dropout regularisation, the cross entropy cost function is no longer easy to define due to all the intermediate probabilities. Why is this the case? Why doesn't the old definition still hold? As long as the network learns better parameters, won't the cross entropy cost decrease on every iteration of Gradient Descent? Thanks in advance.

• Cross entropy is always defined between two distributions, in this case "predictions" and "true labels" - nothing changes about the expression. You probably need to explain a bit more what you mean by "I have learnt that if we do apply dropout regularisation, the cross entropy cost function is no longer easy to define due to all the intermediate probabilities." to get an answer going beyond this... Oct 25, 2019 at 7:32
• I understand that the expression itself doesn't change. What I mean to ask is, is it sensible to use this particular evaluation of the cost when we use dropout regularisation? Or should the expression be modified to account for the probabilities? I am following deeplearning.ai's lectures on Coursera and Andrew Ng mentions that if we apply dropout, we can't plot the decrease in cost function(J) vs number of iterations of Gradient Descent as J is difficult to define in this case. He doesn't really go into further details, which is why I have posted this question. Oct 25, 2019 at 8:29
• I am not familiar with this particular course, but people use the same cross-entropy loss regardless of dropout. Oct 25, 2019 at 9:31