What are neurons in neural networks / how do they work?

Question

From what I found out until now:

A layer in a neural network consists of a parameterizable number of neurons. A neural network consists of multiple layers. A neuron consists of a function f(x1, x2, ..., xn), a sigmoid function which uses f as input and gives a binary output and a weight factor which is multiplied with with the sigmoid function and determines how much this neuron is considered for the output of the layer.

Is this correct and what is the function f/how does it combine the inputs to the output?

(I tried to read the literature and some course material myself but they all either ignore this part/call it a black box or assume you have a PhD in Theoretical Math but my university only focused on Applied Informatics. So if you have literature understandable for someone like me I would also appreciate it)

Answer 1

You are correct in your overall view of the subject. The neuron is nothing more than a set of inputs, a set of weights, and an activation function. The neuron translates these inputs into a single output, which can then be picked up as input for another layer of neurons later on.

While details can vary between neural networks, the function $f(x_1, x_2, \ldots , x_n)$ is often just a weighted sum: $$ f(x_1, x_2, \ldots , x_n) = w_1\cdot x_1 + w_2\cdot x_2 + ... + w_n\cdot x_n $$ Each neuron has a weight vector $w = (w_1, w_2, ..., w_n)$, where $n$ is the number of inputs to that neuron. These inputs can be either the 'raw' input features — say temperature, precipitation, and wind speed for a weather model — or the output of neurons from an earlier layer.

The weights for each neuron are turned during the training stage such that the final network output is biased toward some value (usually 1) for signal, and another (usually -1 or 0) for background.

Non-linear behavior in a neural network is accomplished by use of an activation function (often a sigmoid function) to which the output of $f$ is passed and modified. This allows neural networks to describe more complicated systems while still combining inputs in a simple fashion.

I found the Deep Learning in a Nutshell series by Tim Dettmers to be a very approachable introduction to the subject of machine learning in general.

Answer 2

Within an artificial neural network, a neuron is a mathematical function that model the functioning of a biological neuron. Typically, a neuron compute the weighted average of its input, and this sum is passed through a nonlinear function, often called activation function, such as the sigmoid. I attach an image from an AI course that illustrates it (note that in this particular case the weighted sum contains also a bias term). The output of the neuron can then be sent as input to the neurons of another layer, which could repeat the same computation (weighted sum of the input and transformation with activation function).

Note that this computation correspond to multiplying a vector of input/activation states with a matrix of weights (and passing the resulting vector through the activation function).

If you are interested in neural networks, not only from the engineering point of view but also in how these artificial neurons resemble real biological neurons, a classic book which could still be interesting is Parallel Distributed Processing by Rumelhart and McClelland

Answer 3

The simplest versions are fairly simple, especially the regression case. If $y$ is your outcome and $X$is your data, the model is

$$ y= Z'\beta $$ $$ Z=\sigma(X'\alpha) $$ $$ \sigma(x) = 1/(1+e^{-x}) $$

$\alpha$ and $\beta$ are parameters, and are chosen by an optimizer. Fixing the number of Z's is the same as fixing the number of columns of $\alpha$

Answer 4

In Software Engineering Artifical Neural Networks, Neurons are "containers" of mathematical functions, typically drawn as circles in Artificial Neural Networks graphical representations (see picture below). One or more neurons form a layer -- a set of layers typically disposed in vertical line in Artificial Neural Networks representations. In more complex Hardware Systems, each computer, or each cluster of computers, can be seen as a single neuron in graphical representations. In the scope of the Software Engineering point of view:

Neurons can belong to input layers (red circles below), hidden layers (blue circles) or output layers (green circles).

In a simple one-way Artificial Neural Network (also called Feed-Forward Artificial Neural Networks):

• Input Layer Neurons receive the input information (usually numeric representations of text, image, audio and others types of data), process it through a mathematical function (activation function) and "send" an output to the next layer's neurons based in conditions. On the way to other layer's neurons, that data is multiplied by preset weights (placed in the graphical lines linking one neuron to the others).

• Hidden layer neurons receive inputs from the input layers or by the previous hidden layer, pass them through new functions and send the result to the next layer neurons. Again, the data here is typically multiplied by weights on the way.

• Output layer neurons receive inputs from previous layers, process them through new functions and outputs the expected results. The results could be simple binary classifications (0 or 1, yes or no, black or white, dog or not dog), multiple choice classifications (e.i.: cat, dog or wolf), numeric predictions, matrices and so on. Depending on the type of Artificial Neural Network, this output could be used as a final result, or as an output to a new loop over the same or another neural net.

The mathematical function contained in a neuron can vary depending on the type of Artificial Neural Network -- they could be simple regression functions, non-linear sigmoid functions and so on.