Can a unit-step function introduce non-linearity if used in the hidden nodes?

Question

I have a follow up from this question. From the answers in that question, it's obvious that at minimum, a shallow network (only has a single hidden layer), as defined in the "Universal Approximation Theorem," is needed to have any sort of non-linearity. The original perceptron only had input and output nodes and used the unit-step function, hence it was simply a linear classifier.

If I have a shallow network and I use the unit-step function as the activation for the hidden nodes and the output nodes, is the model considered a non-linear classifier? I realize that this activation function is not desirable as it is not continuous/differentiable. But regardless of desirability, will this model be considered non-linear?

In fact, to take it a step further, if I had a shallow network and I used unit-step activation for the hidden nodes, but a linear activation for the output, this would still be considered a model that can (no guarantee that the training algorithm can train the model) solve non-linearly separable problems, is that correct?

Answer 1

If I have a shallow network and I use the unit-step function as the activation for the hidden nodes and the output nodes, is the model considered a non-linear classifier?

Yes, such model could be used as a non-linear classifier.

In fact, to take it a step further, if I had a shallow network and I used unit-step activation for the hidden nodes, but a linear activation for the output, this would still be considered a model that can (no guarantee that the training algorithm can train the model) solve non-linearly separable problems, is that correct?

Such model can (in theory) approximate any mapping to arbitrary precision (given enough units). From Bishop's great book Neural networks for pattern recongition, sec. 4.3.2, p. 131 (emphasis mine):

Thus we see that the function $y(x_1,x_2)$ can be expressed as a linear combination of step functions whose arguments are linear combinations of $x_1$ and $x_2$. In other words the function $y(x_1,x_2)$ can be approximated by a two-layer network with threshold hidden units and linear output units.