What does the hidden layer in a neural network compute?

Question

I'm sure many people will respond with links to 'let me google that for you', so I want to say that I've tried to figure this out so please forgive my lack of understanding here, but I cannot figure out how the practical implementation of a neural network actually works.

I understand the input layer and how to normalize the data, I also understand the bias unit, but when it comes to the hidden layer, what the actual computation is in that layer, and how it maps to the output is just a little foggy. I've seen diagrams with question marks in the hidden layer, boolean functions like AND/OR/XOR, activation functions, and input nodes that map to all of the hidden units and input nodes that map to only a few hidden units each and so I just have a few questions on the practical aspect. Of course, a simple explanation of the entire neural network process like you would explain to a child, would be awesome.

What computations are done in the hidden layer?

How are those computations mapped to the output layer?

How does the ouput layer work? De-normalizing the data from the hidden layer?

Why are some layers in the input layer connected to the hidden layer and some are not?

Answer 1

Three sentence version:

• Each layer can apply any function you want to the previous layer (usually a linear transformation followed by a squashing nonlinearity).

• The hidden layers' job is to transform the inputs into something that the output layer can use.

• The output layer transforms the hidden layer activations into whatever scale you wanted your output to be on.

Like you're 5:

If you want a computer to tell you if there's a bus in a picture, the computer might have an easier time if it had the right tools.

So your bus detector might be made of a wheel detector (to help tell you it's a vehicle) and a box detector (since the bus is shaped like a big box) and a size detector (to tell you it's too big to be a car). These are the three elements of your hidden layer: they're not part of the raw image, they're tools you designed to help you identify busses.

If all three of those detectors turn on (or perhaps if they're especially active), then there's a good chance you have a bus in front of you.

Neural nets are useful because there are good tools (like backpropagation) for building lots of detectors and putting them together.

---

Like you're an adult

A feed-forward neural network applies a series of functions to the data. The exact functions will depend on the neural network you're using: most frequently, these functions each compute a linear transformation of the previous layer, followed by a squashing nonlinearity. Sometimes the functions will do something else (like computing logical functions in your examples, or averaging over adjacent pixels in an image). So the roles of the different layers could depend on what functions are being computed, but I'll try to be very general.

Let's call the input vector $x$, the hidden layer activations $h$, and the output activation $y$. You have some function $f$ that maps from $x$ to $h$ and another function $g$ that maps from $h$ to $y$.

So the hidden layer's activation is $f(x)$ and the output of the network is $g(f(x))$.

Why have two functions ($f$ and $g$) instead of just one?

If the level of complexity per function is limited, then $g(f(x))$ can compute things that $f$ and $g$ can't do individually.

---

An example with logical functions:

For example, if we only allow $f$ and $g$ to be simple logical operators like "AND", "OR", and "NAND", then you can't compute other functions like "XOR" with just one of them. On the other hand, we could compute "XOR" if we were willing to layer these functions on top of each other:

First layer functions:

• Make sure that at least one element is "TRUE" (using OR)
• Make sure that they're not all "TRUE" (using NAND)

Second layer function:

• Make sure that both of the first-layer criteria are satisfied (using AND)

The network's output is just the result of this second function. The first layer transforms the inputs into something that the second layer can use so that the whole network can perform XOR.

---

An example with images:

Slide 61 from this talk--also available here as a single image--shows (one way to visualize) what the different hidden layers in a particular neural network are looking for.

The first layer looks for short pieces of edges in the image: these are very easy to find from raw pixel data, but they're not very useful by themselves for telling you if you're looking at a face or a bus or an elephant.

The next layer composes the edges: if the edges from the bottom hidden layer fit together in a certain way, then one of the eye-detectors in the middle of left-most column might turn on. It would be hard to make a single layer that was so good at finding something so specific from the raw pixels: eye detectors are much easier to build out of edge detectors than out of raw pixels.

The next layer up composes the eye detectors and the nose detectors into faces. In other words, these will light up when the eye detectors and nose detectors from the previous layer turn on with the right patterns. These are very good at looking for particular kinds of faces: if one or more of them lights up, then your output layer should report that a face is present.

This is useful because face detectors are easy to build out of eye detectors and nose detectors, but really hard to build out of pixel intensities.

So each layer gets you farther and farther from the raw pixels and closer to your ultimate goal (e.g. face detection or bus detection).

---

Answers to assorted other questions

"Why are some layers in the input layer connected to the hidden layer and some are not?"

The disconnected nodes in the network are called "bias" nodes. There's a really nice explanation here. The short answer is that they're like intercept terms in regression.

"Where do the "eye detector" pictures in the image example come from?"

I haven't double-checked the specific images I linked to, but in general, these visualizations show the set of pixels in the input layer that maximize the activity of the corresponding neuron. So if we think of the neuron as an eye detector, this is the image that the neuron considers to be most eye-like. Folks usually find these pixel sets with an optimization (hill-climbing) procedure.

In this paper by some Google folks with one of the world's largest neural nets, they show a "face detector" neuron and a "cat detector" neuron this way, as well as a second way: They also show the actual images that activate the neuron most strongly (figure 3, figure 16). The second approach is nice because it shows how flexible and nonlinear the network is--these high-level "detectors" are sensitive to all these images, even though they don't particularly look similar at the pixel level.

---

Let me know if anything here is unclear or if you have any more questions.

Answer 2

I'm going to describe my view of this in two steps: The input-to-hidden step and the hidden-to-output step. I'll do the hidden-to-output step first because it seems less interesting (to me).

Hidden-to-Output

The output of the hidden layer could be different things, but for now let's suppose that they come out of sigmoidal activation functions. So they are values between 0 and 1, and for many inputs they may just be 0's and 1's.

I like to think of the transformation between these hidden neurons' outputs and the output layer as just a translation (in the linguistic sense, not the geometric sense). This is certainly true if the transformation is invertible, and if not then something was lost in translation. But you basically just have the hidden neurons' outputs seen from a different perspective.

Input-to-Hidden

Let's say you have 3 input neurons (just so I can easily write some equations here) and some hidden neurons. Each hidden neuron gets as input a weighted sum of inputs, so for example maybe

hidden_1 = 10 * (input_1) + 0 * (input_2) + 2 * (input_3)

This means that the value of hidden_1 is very sensitive to the value of input_1, not at all sensitive to input_2 and only slightly sensitive to input_3.

So you could say that hidden_1 is capturing a particular aspect of the input, which you might call the "input_1 is important" aspect.

The output from hidden_1 is usually formed by passing the input through some function, so let's say you are using a sigmoid function. This function takes on values between 0 and 1; so think of it as a switch which says that either input_1 is important or it isn't.

So that's what the hidden layer does! It extracts aspects, or features of the input space.

Now weights can be negative too! Which means that you can get aspects like "input_1 is important BUT ALSO input_2 takes away that importance":

hidden_2 = 10 * (input_1) - 10 * (input_2 ) + 0 * (input_3)

or input_1 and input_3 have "shared" importance:

hidden_3 = 5 * (input_1) + 0 * (input_2) + 5 * (input_3)

More Geometry

If you know some linear algebra, you can think geometrically in terms of projecting along certain directions. In the example above, I projected along the input_1 direction.

Let's look at hidden_1 again, from above. Once the value at input_1 is big enough, the output of the sigmoid activation function will just stay at 1, it won't get any bigger. In other words, more and more input_1 will make no difference to the output. Similarly, if it moves in the opposite (i.e. negative) direction, then after a point the output will be unaffected.

Ok, fine. But suppose we don't want sensitivity in the direction of infinity in certain a direction, and we want it to be activated only for a certain range on a line. Meaning for very negative values there is no effect, and for very positive values there is no effect, but for values between say, 5 and 16 you want it to wake up. This is where you would use a radial basis function for your activation function.

Summary

The hidden layer extracts features of the input space, and the output layer translates them into the desired context. There may be much more to it than this, what with multi-layer networks and such, but this is what I understand so far.

EDIT: This page with its wonderful interactive graphs does a better job than my long and cumbersome answer above could ever do: http://neuralnetworksanddeeplearning.com/chap4.html

Answer 3

I'll try to add to the intuitive operational description...

A good intuitive way to think about a neural network is to think about what a linear regression model attempts to do. A linear regression will take some inputs and come up with a linear model which takes each input value times some model optimal weighting coefficients and tries to map the sum of those results to an output response that closely matches the true output. The coefficients are determined by finding the values which will minimize some error metric between the desired output value and the value that is learned by the model. Another way to say it is that the linear model will try to create coefficient multipliers for each input and sum all of them to try to determine the relationship between the (multiple) input and (typically single) output values. That same model can almost be thought of as the basic building block of a neural network; a single unit perceptron.

But the single unit perceptron has one more piece that will process the sum of the weighted data in a non-linear manner. It typically uses a squashing function (sigmoid, or tanh) to accomplish this. So you have the basic unit of the hidden layer, which is a block that will sum a set of weighted inputs-- it then passes the summed response to a non-linear function to create an (hidden layer) output node response. The bias unit is just as in linear regression, a constant offset which is added to each node to be processed. Because of the non-linear processing block, you are no longer limited to linear only responses (as in the linear regression model).

Ok, but when you have many of the single perceptron units working together, each can have different input weight multipliers and different responses (even though ALL process the same set of inputs with the same non-linear block previously described). What makes the responses different is that each has different coefficient weights that are learned by the neural network via training (some forms include gradient descent). The result of all of the perceptrons are then processed again and passed to an output layer, just as the individual blocks were processed. The question then is how are the correct weights determined for all of the blocks?

A common way to learn the correct weights is by starting with random weights and measuring the error response between the true actual output and the learned model output. The error will typically get passed backwards through the network and the feedback algorithm will individually increase or decrease those weights by some proportion to the error. The network will repeatedly iterate by passing forward, measuring the output response, then updating (passing backwards weight adjustments) and correcting the weights until some satisfactory error level is reached. At that point you have a regression model that can be more flexible than a linear regression model, it is what is commonly called a universal function approximator.

One of the ways that really helped me to learn how a neural network truly operates is to study the code of a neural network implementation and build it. One of the best basic code explanations can be found in the neural network chapter of (the freely available) 'The Scientist and Engineer's guide to DSP' Ch. 26. It is mostly written in very basic language (I think it was fortran) that really helps you to see what is going on.

Answer 4

Let us take the case of classification. What the output layer is trying to do is estimate the conditional probability that your sample belongs to a given class, i.e. how likely is for that sample to belong to a given class. In geometrical terms, combining layers in a non-linear fashion via the threshold functions allows the neural networks to solve non-convex problems (speech recognition, object recognition, and so on), which are the most interesting ones. In other words, the output units are able to generate non-convex decision functions like those depicted here.

One can view the units in hidden layers as learning complex features from data that allow the output layer to be able to better discern one class from another, to generate more acurate decision boundaries. For example, in the case of face recognition, units in the first layers learn edge like features (detect edges at given orientations and positions) and higher layer learn to combine those to become detectors for facial features like the nose, mouth or eyes. The weights of each hidden unit represent those features, and its output (assuming it is a sigmoid) represents the probability that that feature is present in your sample.

In general, the meaning of the outputs of output and hidden layers depend on the problem you are trying to solve (regression, classification) and the loss function you employ (cross entropy, least squared errors, ...)