# simple ANN as a set of linear transformations

We cannot classify the points of the XOR problem with a single perceptron in the hidden layer. However, we can achieve this by using two perceptrons in the hidden layer and one for the output layer, without using an activation function. The input layer consists of two perceptrons. With one hidden layer and one output layer, there are two weight matrices: the first matrix $$A$$ is $$2*2$$, and the second matrix $$B$$ is $$1*2$$. Let's consider $$A=[[1,1],[0,1]]$$ and $$B=[-2,2]$$. During forward propagation, we perform a linear transformation of the $$X$$ vector by matrix $$A$$, resulting in a new vector in $$R²$$. This new vector then undergoes a linear transformation by matrix $$B$$ to produce the output. This means that all we need to do is $$B(AX)$$. Here is where my problem arises: Upon researching, it becomes clear that matrix multiplication is associative, meaning $$B(AX)=(BA)X$$,also the combination of two linear transformation is a linear transformation. The combination of $$A$$ and $$B$$ results in another matrix denoted as $$K$$ with a shape of 1x2. Therefore, $$B(AX)=KX$$. The key here is to learn $$K$$ ; initially, we achieve it by working on individual linear transformations, but we can make the artificial neural network (ANN) do this in one step by choosing only one perceptron in the hidden layer. However, this approach is incorrect. So, where is my problem?!!".

I think your specific problem is:

However, we can achieve this by using two perceptrons in the hidden layer and one for the output layer, without using an activation function

If you write out the full calculations showing what your proposed network would return with inputs (0, 0), (0, 1), (1, 0), and (1, 1) you will see that this statement is not true to begin with.

Edit Calculations are below.

(1, 0) done in two steps:

$$\begin{bmatrix}-2 & 2\end{bmatrix} \times \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix} \times \begin{bmatrix}{1 \\ 0}\end{bmatrix} = \begin{bmatrix}-2 & 2\end{bmatrix} \times \begin{bmatrix}1 \\ 0\end{bmatrix} = -2$$

(0, 1) done in two steps:

$$\begin{bmatrix}-2 & 2\end{bmatrix} \times \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix} \times \begin{bmatrix}{0 \\ 1}\end{bmatrix} = \begin{bmatrix}-2 & 2\end{bmatrix} \times \begin{bmatrix}1 \\ 1\end{bmatrix} = 0$$

The answers above should be linearly separable from the ones below.

(0, 0):

$$\begin{bmatrix}-2 & 2\end{bmatrix} \times \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix} \times \begin{bmatrix}{0 \\ 0}\end{bmatrix} = \begin{bmatrix}-2 & 2\end{bmatrix} \times \begin{bmatrix}0 \\ 0\end{bmatrix} = 0$$

And there is no more need to do more - (0,1) and (0, 0) map to the same point in this set-up. And collapsing the two transformation matrices doesn't change anything on that end:

$$\begin{bmatrix}-2 & 2\end{bmatrix} \times \begin{bmatrix}1 & 1 \\ 0 & 1\end{bmatrix} = \begin{bmatrix}-2 & 0\end{bmatrix}$$

(1,0) done in one step:

$$\begin{bmatrix}-2 & 0\end{bmatrix} \times \begin{bmatrix}1 \\ 0\end{bmatrix} = -2$$

(0, 1) done in one step:

$$\begin{bmatrix}-2 & 0\end{bmatrix} \times \begin{bmatrix}0 \\ 1\end{bmatrix} = 0$$

• if you do the calculation you will see that:matrixes that i presented solve this problem by transform the o's labels point to the point -2 in 1D and the 1's labels to the point 0
– Mag
Commented Mar 5 at 22:38
• I think you are wrong. BA = [-2, 0]. The XOR problem requires to have inputs (0,0) and (1,1) with one label, and (1,0) and (0,1) with another one. Your setup sends (0,0) and (0,1) to 0 and (1,0) and (1,1) to -2. but this is not the XOR problem. Commented Mar 6 at 9:20
• yh that is the problem when use the composition of both transformation at once,,if we calculate the first transformation and apply on it the second transf then we get the desired result.and that is my problem i don't know why when doing that we get the desired result but applying the composition of transformation at one step give us wrong results
– Mag
Commented Mar 6 at 12:01
• @Mag I've updated my answer with the calculations that show these matrices don't map XOR properly. Commented Mar 6 at 13:04

You cannot achieve this using multiple layers and more perceptrons without non-linear activation functions. As you noticed, if no activation is used, any dense network, no matter how many layers and no matter how many perceptrons on each layer you use reduces to a simple linear function, which is the equivalent of a single perceptron with no activation function. Since the linear function is a hyperplane, you cannot solve XOR problem since that is not linearly separable.

• but the matrixes that i presented solve this problem by transform the o's labels point to the point -2 in 1D and the 1's labels to the point 0
– Mag
Commented Mar 5 at 22:37
• 0 labels are (0,0) and (1,1), 1 labels are (1,0), (0,1). This is the XOR problem. Your network sends (0,0) and (0,1) to the same label, and (1,0), (1,1) to another label, however this is not the XOR problem. Commented Mar 6 at 9:22
• yh that is the problem when use the composition of both transformation at once,,if we calculate the first transformation and apply on it the second transf then we get the desired result.and that is my problem i don't know why when doing that we get the desired result but applying the composition of transformation at one step give us wrong results
– Mag
Commented Mar 6 at 12:13