2
$\begingroup$

I am learning about ELM (Extreme Learning Machines) and it appears to have no bias weights at the output layer. Besides that, just to clarify, the kind of ELM I am refering to are topologically no different than 2-layer MLPs (or 3-layer if you count the input as a layer).

So, I am wondering whether I understood wrong about the absence of output layer bias weights in ELMs or an MLP can "survive" without it.

$\endgroup$

1 Answer 1

3
$\begingroup$

The bias at output layer is highly recommended if the activation function is Sigmoid. Note that in ELM the activation function at output layer is linear, which indicates the bias is not that required. Suppose there is only one output node, and you add a bias weight at output layer, that will be equivalent to a constant added to the weighted linear combination of g functions shown in Huang's paper. And the matrix expression in (3) is: $H\beta + b = T$, where b = rand(1,1)*ones(1,length(hiddenLayerNode));. It is not hard to observe that the minimum norm of least squared solution will retain with their proposed pseudo-inverse of $H$.

If you are using their Matlab code for some test, you may notice that they suggest scale the input data onto the range of [-1,1]. Within that range, even Sigmoid function at output layer can approximate to a linear function. Thus no bias at that layer is required as well.

I modified their code by adding the bias at the output layer for a sinc function regression, and the performance is almost the same as it is without the bias at output layer. You may also try some other functions, as well as the classification problems, to validate by yourself.

$\endgroup$
4
  • $\begingroup$ Cool that somebody knows about ELM. I adapted the Java version, which is based on the Matlab version, to Scala. I will keep the [-1;1] transform at the output and feed a sigmoid with it. I have seen some discussion about softmax activation functions for generating posteriors probabilities in multiclass problems; Is it worth the trouble or I can just normalize the output? I work with Active Learning, so prob. distributions are an important measure of certainty. $\endgroup$
    – dawid
    Jan 12, 2014 at 18:03
  • 1
    $\begingroup$ @davips, the [-1:1] normalization is performed at the input in ELM, but yes you also need to normalize the output because a uni-polar sigmoid function ranges at (0,1). And logistic regression is a good approximation of bayesian prob, you can use softmax for multiclass problems. You can also use one-vs-all strategy to treat multiclass problem as a two class problem. $\endgroup$
    – lennon310
    Jan 12, 2014 at 18:12
  • $\begingroup$ I meant to use ELM with a softmax output node. In a problem with N-classes, I don't think it is possible, since ELM needs N output nodes while softmax is a unique node that plays the role of all outputs. $\endgroup$
    – dawid
    Jan 17, 2014 at 1:00
  • $\begingroup$ @viyps while softmax is a unique node that plays the role of all outputs, this should not be. What do you mean by this? $\endgroup$
    – Narfanar
    Dec 28, 2018 at 15:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.