Are bias weights essential in the output layer, if one wants a universal function approximator (or non-linearly separable problem solver)? I am learning about ELM (Extreme Learning Machines) and it appears to have no bias weights at the output layer.
Besides that, just to clarify, the kind of ELM I am refering to are topologically no different than 2-layer MLPs (or 3-layer if you count the input as a layer).
So, I am wondering whether I understood wrong about the absence of output layer bias weights in ELMs or an MLP can "survive" without it.
 A: The bias at output layer is highly recommended if the activation function is Sigmoid. Note that in ELM the activation function at output layer is linear, which indicates the bias is not that required. Suppose there is only one output node, and you add a bias weight at output layer, that will be equivalent to a constant added to the weighted linear combination of g functions shown in Huang's paper. And the matrix expression in (3) is:
 $H\beta + b = T$, where b = rand(1,1)*ones(1,length(hiddenLayerNode));. It is not hard to observe that the minimum norm of least squared solution will retain with their proposed pseudo-inverse of $H$.
If you are using their Matlab code for some test, you may notice that they suggest scale the input data onto the range of [-1,1]. Within that range, even Sigmoid function at output layer can approximate to a linear function. Thus no bias at that layer is required as well.
I modified their code by adding the bias at the output layer for a sinc function regression, and the performance is almost the same as it is without the bias at output layer. You may also try some other functions, as well as the classification problems, to validate by yourself.
