I am using feed-forward NN. I understand the concept, but my question is about weights. How can you interpret them, i.e. what do they represent or how can they be undestrood (besied just function coefficients)? I have found something called "space of weights", but I am not quite sure what does it means.
Individual weights represent the strength of connections between units. If the weight from unit A to unit B has greater magnitude (all else being equal), it means that A has greater influence over B (i.e. to increase or decrease B's level of activation).
You can also think of the the set of incoming weights to a unit as measuring what that unit 'cares about'. This is easiest to see at the first layer. Say we have an image processing network. Early units receive weighted connections from input pixels. The activation of each unit is a weighted sum of pixel intensity values, passed through an activation function. Because the activation function is monotonic, a given unit's activation will be higher when the input pixels are similar to the incoming weights of that unit (in the sense of having a large dot product). So, you can think of the weights as a set of filter coefficients, defining an image feature. For units in higher layers (in a feedforward network), the inputs aren't from pixels anymore, but from units in lower layers. So, the incoming weights are more like 'preferred input patterns'.
Not sure about your original source, but if I were talking about 'weight space', I'd be referring to the set of all possible values of all weights in the network.
Well, it depends on a network architecture and particular layer. In general NNs are not interpretable, this is their major drawback in commercial data analysis (where your goal is to uncover actionable insights from your model).
But I love convolutional networks, because they are different! Although their upper layers learn very abstract concepts, usable for transfer learning and classification, which could not be understood easily, their bottom layers learn Gabor filters straight from raw data (and thus are interpretable as such filters). Take a look at the example from a Le Cun lecture:
In addition, M. Zeiler (pdf) and many other researchers invented very creative method to "understand" convnet and ensure it learned something useful dubbed Deconvolutional networks, in which they 'trace' some convnet by making forward pass over input pictures and remembering which neurons had largest activations for which pics. This gives stunning introspection like this (a couple of layers were shown below):
Gray images at the left side are neuron activations (the more intensity -- the larger activation) by color pictures at the right side. We see, that these activations are skeletal representations of real pics, i.e., the activations are not random. Thus, we have a solid hope, that our convnet indeed learned something useful and will have decent generalization in unseen pics.
I think you are trying too hard on the model that does not have too much interpretability. Neural network (NN) is one of the black box models that will give you better performance, but it is hard to understand what was going on inside. Plus, it is very possible to have thousands even millions of weights inside of NN.
NN is a very big non-linear non-convex function that can have large amount of local minima. If you train it multiple times, with different start point, the weights will be different. You can come up with some ways to visualize the internal weights, but it also does not give you too much insights.
Here is one example on NN visualization for MNIST data. The upper right figure (reproduced below) shows the transformed features after applying the weights.