Can a unit-step function introduce non-linearity if used in the hidden nodes? I have a follow up from this question. From the answers in that question, it's obvious that at minimum, a shallow network (only has a single hidden layer), as defined in the "Universal Approximation Theorem," is needed to have any sort of non-linearity. The original perceptron only had input and output nodes and used the unit-step function, hence it was simply a linear classifier. 
If I have a shallow network and I use the unit-step function as the activation for the hidden nodes and the output nodes, is the model considered a non-linear classifier? I realize that this activation function is not desirable as it is not continuous/differentiable. But regardless of desirability, will this model be considered non-linear?
In fact, to take it a step further, if I had a shallow network and I used unit-step activation for the hidden nodes, but a linear activation for the output, this would still be considered a model that can (no guarantee that the training algorithm can train the model) solve non-linearly separable problems, is that correct?
 A: 
If I have a shallow network and I use the unit-step function as the activation for the hidden nodes and the output nodes, is the model considered a non-linear classifier?

Yes, such model could be used as a non-linear classifier.

In fact, to take it a step further, if I had a shallow network and I used unit-step activation for the hidden nodes, but a linear activation for the output, this would still be considered a model that can (no guarantee that the training algorithm can train the model) solve non-linearly separable problems, is that correct?

Such model can (in theory) approximate any mapping to arbitrary precision (given enough units). From Bishop's great book Neural networks for pattern recongition, sec. 4.3.2, p. 131 (emphasis mine):

Thus we see that  the  function   $y(x_1,x_2)$   can  be expressed  as  a  linear  combination  of  step  functions  whose  arguments  are  linear  combinations  of  $x_1$   and  $x_2$.  In  other  words  the  function  $y(x_1,x_2)$     can  be  approximated   by  a  two-layer  network  with  threshold   hidden  units  and   linear   output  units. 

