I have a follow up from this question. From the answers in that question, it's obvious that at minimum, a shallow network (only has a single hidden layer), as defined in the "Universal Approximation Theorem," is needed to have any sort of non-linearity. The original perceptron only had input and output nodes and used the unit-step function, hence it was simply a linear classifier.
If I have a shallow network and I use the unit-step function as the activation for the hidden nodes and the output nodes, is the model considered a non-linear classifier? I realize that this activation function is not desirable as it is not continuous/differentiable. But regardless of desirability, will this model be considered non-linear?
In fact, to take it a step further, if I had a shallow network and I used unit-step activation for the hidden nodes, but a linear activation for the output, this would still be considered a model that can (no guarantee that the training algorithm can train the model) solve non-linearly separable problems, is that correct?