# What is the correct way of calculating Rectifier Linear and MaxOut functions?

If I have a artificial neuron with 2 inputs:

input 1 = 0.7 & weight = 0.7
input 2 = 0.3 & weight = 0.3


If I use a Rectifier Linear (ReLU) as activation function ($f(x)=max(0,x)$) is the output of this neuron:

Outcome 1: {I assume this is correct}: ReLU = $max(0,0.7*0.7+0.3*0.3) = 0.58$

or

Outcome 2: ReLU = $max(0,0.7*0.7,0.3*0.3) = 0.49$

Or in other words: is this max taken over the summation or individual inputs? As I see ReLU as a drop in replacement for sigmoid/tanh I expect the first one is correct.

Question 1: is outcome 1 or 2 the correct way of calculating ReLU

I started to hesistate because of maxout which is defined in the paper as:

$h_i = \max_{j \in [1, k]} z_{ij}$ with $z_{ij} = x^T W_{ij} + b_{ij}$, $W \in \mathbb{R}^{d \times m \times k}$ and $b \in \mathbb{R}^{m \times k}$

Based on this definition I expect that the output of this neuron if maxout is used as activation is: $max(0.7*0.7,0.3*0.3) = 0.49$

Question 2: is this the correct way of calculating maxout for 1 layer with 1 neuron with these 2 input/weights??

The reason why I hesistate about the maxout/ReLU algo impls is because of the following phrase in this paper:

The only difference between maxout and max pooling over a set of rectified linear units is that maxout does not include a 0 in the max.

This suggests (to me) that $ReLU == MaxOut$ if the maximum $value > 0$ so that outcome 2 is correct where I first (still) assume outcome 1 is correct.

If I have a artificial neuron with 2 inputs:

input 1 = 0.7 & weight = 0.7
input 2 = 0.3 & weight = 0.3


Let's make things clear. Consider the non-spatial case where activation function's input is just a 1D vector: $\vec{x} = (x_1, ..., x_d)$ (where do your weights come from?), which is usually an output of a Linear or Batch Normalization layer but can be what you want.

Typical activation functions are $\mathbb{R} \rightarrow \mathbb{R}$. That means they handle each input component individually. For example, ReLU:

$ReLU_i(x_i) = max(0, x_i)$; Note that no weight involved.

Applied to an input vector: $ReLU((1, -2, 3, -4, 5)) = (1, 0, 3, 0, 5)$

But MaxOut is a $\mathbb{R}^d \rightarrow \mathbb{R}$ function:

$MaxOut_s(\vec{x}) = max( \vec{w}_{s1} \vec{x} + b_{s1}, ... , \vec{w}_{sk} \vec{x} + b_{sk})$

$MaxOut(\vec{x}) = (MaxOut_1(\vec{x}), ..., MaxOut_m(\vec{x}))$

where the number of components $k$ in each MaxOut function and the amount $m$ of such functions are up to you, $w_{sl} \in \mathbb{R}^d, b_{sl} \in \mathbb{R}$ are learned.

• The weights are not necessary needed for this example, but typically the output of a neuron is calculated by f(sum of all (weights * input)), where then f is the activiation function. – robert Sep 18 '15 at 5:53
• @jos Yes, classical neuron is a composition (Activation ∘ Linear)(input). In practice, however, we separate these two parts as they are completely independent and have nothing to do with the internal structure of each other (activation function doesn't see linear layer's weights). – Alexander Lutsenko Sep 18 '15 at 9:23