simple ANN as a set of linear transformations

Question

We cannot classify the points of the XOR problem with a single perceptron in the hidden layer. However, we can achieve this by using two perceptrons in the hidden layer and one for the output layer, without using an activation function. The input layer consists of two perceptrons. With one hidden layer and one output layer, there are two weight matrices: the first matrix $A$ is $2*2$, and the second matrix $B$ is $1*2$. Let's consider $A=[[1,1],[0,1]]$ and $B=[-2,2]$. During forward propagation, we perform a linear transformation of the $X$ vector by matrix $A$, resulting in a new vector in $R²$. This new vector then undergoes a linear transformation by matrix $B$ to produce the output. This means that all we need to do is $B(AX)$. Here is where my problem arises: Upon researching, it becomes clear that matrix multiplication is associative, meaning $B(AX)=(BA)X$,also the combination of two linear transformation is a linear transformation. The combination of $A$ and $B$ results in another matrix denoted as $K$ with a shape of 1x2. Therefore, $B(AX)=KX$. The key here is to learn $K$ ; initially, we achieve it by working on individual linear transformations, but we can make the artificial neural network (ANN) do this in one step by choosing only one perceptron in the hidden layer. However, this approach is incorrect. So, where is my problem?!!".

Answer 1

I think your specific problem is:

However, we can achieve this by using two perceptrons in the hidden layer and one for the output layer, without using an activation function

If you write out the full calculations showing what your proposed network would return with inputs (0, 0), (0, 1), (1, 0), and (1, 1) you will see that this statement is not true to begin with.

Edit Calculations are below.

(1, 0) done in two steps:

$$ \begin{bmatrix}-2 & 2\end{bmatrix} \times \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix} \times \begin{bmatrix}{1 \\ 0}\end{bmatrix} = \begin{bmatrix}-2 & 2\end{bmatrix} \times \begin{bmatrix}1 \\ 0\end{bmatrix} = -2 $$

(0, 1) done in two steps:

$$ \begin{bmatrix}-2 & 2\end{bmatrix} \times \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix} \times \begin{bmatrix}{0 \\ 1}\end{bmatrix} = \begin{bmatrix}-2 & 2\end{bmatrix} \times \begin{bmatrix}1 \\ 1\end{bmatrix} = 0 $$

The answers above should be linearly separable from the ones below.

(0, 0):

$$ \begin{bmatrix}-2 & 2\end{bmatrix} \times \begin{bmatrix} 1 & 1 \\ 0 & 1 \end{bmatrix} \times \begin{bmatrix}{0 \\ 0}\end{bmatrix} = \begin{bmatrix}-2 & 2\end{bmatrix} \times \begin{bmatrix}0 \\ 0\end{bmatrix} = 0 $$

And there is no more need to do more - (0,1) and (0, 0) map to the same point in this set-up. And collapsing the two transformation matrices doesn't change anything on that end:

$$ \begin{bmatrix}-2 & 2\end{bmatrix} \times \begin{bmatrix}1 & 1 \\ 0 & 1\end{bmatrix} = \begin{bmatrix}-2 & 0\end{bmatrix} $$

(1,0) done in one step:

$$ \begin{bmatrix}-2 & 0\end{bmatrix} \times \begin{bmatrix}1 \\ 0\end{bmatrix} = -2 $$

(0, 1) done in one step:

$$ \begin{bmatrix}-2 & 0\end{bmatrix} \times \begin{bmatrix}0 \\ 1\end{bmatrix} = 0 $$

Answer 2

You cannot achieve this using multiple layers and more perceptrons without non-linear activation functions. As you noticed, if no activation is used, any dense network, no matter how many layers and no matter how many perceptrons on each layer you use reduces to a simple linear function, which is the equivalent of a single perceptron with no activation function. Since the linear function is a hyperplane, you cannot solve XOR problem since that is not linearly separable.