# What are neurons in neural networks / how do they work?

From what I found out until now:

A layer in a neural network consists of a parameterizable number of neurons. A neural network consists of multiple layers. A neuron consists of a function f(x1, x2, ..., xn), a sigmoid function which uses f as input and gives a binary output and a weight factor which is multiplied with with the sigmoid function and determines how much this neuron is considered for the output of the layer.

Is this correct and what is the function f/how does it combine the inputs to the output?

(I tried to read the literature and some course material myself but they all either ignore this part/call it a black box or assume you have a PhD in Theoretical Math but my university only focused on Applied Informatics. So if you have literature understandable for someone like me I would also appreciate it)

You are correct in your overall view of the subject. The neuron is nothing more than a set of inputs, a set of weights, and an activation function. The neuron translates these inputs into a single output, which can then be picked up as input for another layer of neurons later on.

While details can vary between neural networks, the function $f(x_1, x_2, \ldots , x_n)$ is often just a weighted sum: $$f(x_1, x_2, \ldots , x_n) = w_1\cdot x_1 + w_2\cdot x_2 + ... + w_n\cdot x_n$$ Each neuron has a weight vector $w = (w_1, w_2, ..., w_n)$, where $n$ is the number of inputs to that neuron. These inputs can be either the 'raw' input features — say temperature, precipitation, and wind speed for a weather model — or the output of neurons from an earlier layer.

The weights for each neuron are turned during the training stage such that the final network output is biased toward some value (usually 1) for signal, and another (usually -1 or 0) for background.

Non-linear behavior in a neural network is accomplished by use of an activation function (often a sigmoid function) to which the output of $f$ is passed and modified. This allows neural networks to describe more complicated systems while still combining inputs in a simple fashion.

I found the Deep Learning in a Nutshell series by Tim Dettmers to be a very approachable introduction to the subject of machine learning in general.

• Wow, I didn't expect it to something as simple as a weighted sum function since we used more complex functions when programming en.wikipedia.org/wiki/Genetic_algorithm in school. – H. Idden Oct 24 '16 at 11:39

Within an artificial neural network, a neuron is a mathematical function that model the functioning of a biological neuron. Typically, a neuron compute the weighted average of its input, and this sum is passed through a nonlinear function, often called activation function, such as the sigmoid. I attach an image from an AI course that illustrates it (note that in this particular case the weighted sum contains also a bias term). The output of the neuron can then be sent as input to the neurons of another layer, which could repeat the same computation (weighted sum of the input and transformation with activation function).

Note that this computation correspond to multiplying a vector of input/activation states with a matrix of weights (and passing the resulting vector through the activation function).

If you are interested in neural networks, not only from the engineering point of view but also in how these artificial neurons resemble real biological neurons, a classic book which could still be interesting is Parallel Distributed Processing by Rumelhart and McClelland

The simplest versions are fairly simple, especially the regression case. If $y$ is your outcome and $X$is your data, the model is

$$y= Z'\beta$$ $$Z=\sigma(X'\alpha)$$ $$\sigma(x) = 1/(1+e^{-x})$$

$\alpha$ and $\beta$ are parameters, and are chosen by an optimizer. Fixing the number of Z's is the same as fixing the number of columns of $\alpha$

• What is Z′β? If I remember my Math lessons correctly, Z' was Z derived by x. σ was the standard deviation in statistics. So does it mean that the standard derivation of input X derived by x (how?) multiplied an optimizer chosen constant α? 1/(1+e−x) was the logistic regression curve when estimating the growth of something with a limit of growth but it also seems to be called sigmoid function at neural networks. Those were the kind of parts where papers lost me. Could you please expand the answer by explaining the formulas so it is understandable for people who only know high school level math? – H. Idden Oct 24 '16 at 11:33
• No, the apostrophes indicate matrix transposes. $Z_1\beta_1+Z_2\beta_2...$ – generic_user Oct 28 '16 at 0:49
• Oh thanks, we used deriving for 4 years in Math but matrices only for 2 months. Makes sense with when you see each layer as a matrix. – H. Idden Oct 29 '16 at 12:46