What did my neural network just learn? What features does it care about and why? A neural net learns features of a data set as a means of achieving some goal.  When it is done, we may want to know what the neural net learned.  What were the features and why did it care about those.  Can someone give some references on the body of work that concerns this problem?
 A: Neural Network is one of the black box models that would not give "easy to understand" rules / or what has been learned.
Specifically, what has been learned are the parameters in the model, but the parameters can be large: hundreds of thousands of the parameters is very normal. 
In addition, it is also not clear on the important features learned, you can understand the model uses all the features, with many complicated operations to derive the results, where not easy to say in plain English how the model transform each feature use it.

In fact, the one layer neural network (without hidden layer) with logistic function as activation function is identical to logistic regression. Logistic regression is very rich in interpretations. Here is one example. But with complex neural network / more hidden layers, such interpretation will not apply.
A: It is true that it's hard to understand what a neural network is learning but there has been a lot of work on that front. We definitely can get some idea of what our network is looking for.
Let's consider the case of a convolutional neural net for images. We have the interpretation for our first layer that we are sliding $K$ filters over the image, so our first hidden layer corresponds to the agreement between small chunks of the image and our various filters. We can visualize these filters to see what our first layer of representation is:

This picture is of the first layer of filters from an AlexNet and is taken from this wonderful tutorial: http://cs231n.github.io/understanding-cnn/. This lets us interpret the first hidden layer as learning to represent the image, consisting of raw pixels, as a tensor where each coordinate is the agreement of a filter with a small region of the image. The next layer then is working with these filter activations. 
It's not so hard to understand the first hidden layer because we can just look at the filters to see how they behave, because they're directly applied to an input image. E.g. let's say you're working with a black and white image (so our filters are 2D rather than 3D) and you have a filter that's something like
$$
\begin{bmatrix}0 & 1 & 0 \\ 1 & -4 & 1 \\ 0 & 1 & 0\end{bmatrix}.
$$
Imagine applying this to a 3x3 region of an image (ignoring the bias term). If every pixel was the same color then you'd get $0$ since they'd cancel out. But if the upper half is different from the lower half, say, then you'll get a potentially large value. This filter, in fact, is an edge detector, and we can figure that out by actually just applying it to images and seeing what happens. But it's a lot harder to understand the deeper layers because the whole problem is we don't know how to interpret what we're applying the filters to.
This paper by Erhan et al (2009) agrees with this: they say that first hidden layer visualizations are common (and that was back in 2009) but visualizing the deeper layers is the hard part. From that paper:

The main experimental finding of this investigation is very
  surprising: the response of an internal unit to input images, as a
  function in image space, appears to be unimodal, or at least that the
  maximum is found reliably and consistently for all the random
  initializations tested. This is interesting because finding this
  dominant mode is relatively easy, and displaying it then provides a
  good characterization of what the unit does.

Chris Olah et al (https://distill.pub/2017/feature-visualization/) build on this and discuss how in general you can (1) generate images that lead to large activations in order to get a sense of what the network is looking for; or (2) take actual input images and see how different parts of the image activate the network. That post focuses on (1).
In the image below, taken from that linked article by Olah et al., the authors discuss the different aspects of the network that you can inspect. The left-most image shows the result of optimizing the activation of a particular neuron over the input image space, and so on. 

I would highly recommend reading that article in its entirety if you want a deeper understanding of this, and by reading its references you should have a great grasp of what's been done with this.
Now of course this was all just for images where we as humans can make sense of the inputs. If you're working with something harder to interpret, like just a big vector of numbers, then you may not be able to make such cool visualizations, but in principle you could still consider these techniques for assessing the various neurons, layers, and etc.
