17
$\begingroup$

I've been thinking about, implementing and using the Extreme Learning Machine (ELM) paradigm for more than a year now, and the longer I do, the more I doubt that it is really a good thing. My opinion, however, seems to be in contrast with scientific community where -- when using citations and new publications as a measure -- it seems to be a hot topic.

The ELM has been introduced by Huang et. al. around 2003. The underlying idea is rather simple: start with a 2-layer artificial neural network and randomly assign the coefficients in the first layer. This, one transforms the non-linear optimization problem which is usually handled via backpropagation into a simple linear regression problem. More detailed, for $\mathbf x \in \mathbb R^D$, the model is

$$ f(\mathbf x) = \sum_{i=1}^{N_\text{hidden}} w_i \, \sigma\left(v_{i0} + \sum_{k=1}^{D} v_{ik} x_k \right)\,.$$

Now, only the $w_i$ are adjusted (in order to minimize squared-error-loss), whereas the $v_{ik}$'s are all chosen randomly. As a compensation for the loss in degrees-of-freedom, the usual suggestion is to use a rather large number of hidden nodes (i.e. free parameters $w_i$).

From another perspective (not the one usually promoted in the literature, which comes from the neural network side), the whole procedure is simply linear regression, but one where you choose your basis functions $\phi$ randomly, for example

$$ \phi_i(\mathbf x) = \sigma\left(v_{i0} + \sum_{k=1}^{D} v_{ik} x_k \right)\,.$$

(Many other choices beside the sigmoid are possible for the random functions. For instance, the same principle has also been applied using radial basis functions.)

From this viewpoint, the whole method becomes almost too simplistic, and this is also the point where I start to doubt that the method is really a good one (... whereas its scientific marketing certainly is). So, here are my questions:

  • The idea to raster the input space using random basis functions is, in my opinion, good for low dimensions. In high dimensions, I think it is just not possible to find a good choice using random selection with a reasonable number of basisfunctions. Therefore, does the ELM degrade in high-dimensions (due to the curse of dimensionality)?

  • Do you know of experimental results supporting/contradicting this opinion? In the linked paper there is only one 27-dimensional regression data set (PYRIM) where the method performs similar to SVMs (whereas I would rather like to see a comparison to a backpropagation ANN)

  • More generally, I would like to here your comments about the ELM method.

$\endgroup$
9
$\begingroup$

Your intuition about the use of ELM for high dimensional problems is correct, I have some results on this, which I am preparing for publication. For many practical problems, the data are not very non-linear and the ELM does fairly well, but there will always be datasets where the curse of dimensionality means that the chance of finding a good basis function with curvature just where you need it becomes rather small, even with many basis vectors.

I personally would use something like a least-squares support vector machine (or a radial basis function network) and try and choose the basis vectors from those in the training set in a greedy manner (see e.g. my paper, but there were other/better approaches that were published at around the same time, e.g. in the very good book by Scholkopf and Smola on "Learning with Kernels"). I think it is better to compute an approximate solution to the exact problem, rather than an exact solution to an approximate problem, and kernel machines have a better theoretical underpinning (for a fixed kernel ;o).

$\endgroup$
  • 1
    $\begingroup$ +1. I have never heard about ELM before, but from the description in the OP it sounds a bit like liquid state machine (LSM): random network connectivity and optimizing only the readout weights. However, in LSM the random "reservoir" is recurrent, whereas in ELM it is feedforward. Is that indeed the similarity and the difference? $\endgroup$ – amoeba Feb 11 '15 at 22:13
  • 1
    $\begingroup$ Thank you for the good answer, please update the answer when your paper has been published. Regarding kernel: of course you there's also a "kernel" version of ELM. Just replace the sigmoid above by some (not-necessarily positive-definite) kernel $k(\mathbf x,\mathbf x_i)$ and pick a lot of $\mathbf x_i$'s randomly. Same "trick" here as in the original ELM, same problem. Those methods you mentioned for choosing the centers are of direct importance here as well (even if the goal function in ELM and SVM are different) ... this probably turns it from a "totally blind" to a "half blind" method. $\endgroup$ – davidhigh Feb 14 '15 at 1:54
  • $\begingroup$ @amoeba: I didn't knew the liquid state machine, but from what you say it sounds indeed very similar ... and of course, techinically more general. Still, the recurrency just adds a more complex form of randomness to the problem, which in my opinion doesn't cure the the curse-of-dimensionality problems (... but ok, who does this?). Are those recurrency weights chosen with some care or also completely random? $\endgroup$ – davidhigh Feb 14 '15 at 2:04
  • $\begingroup$ @davidhigh for an RBF kernel, the "representer theorems" show that there is no better solution than to centre a basis function on each training sample (making some reasonable assumptions about the regularised cost function). This is one of the nice features of kernel methods (and splines), so there is no need to spread them randomly. By the way, constructing a linear model on the output of randomly selected basis functions has a very long history, my favourite is the single layer look up perceptron ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=51949&tag=1 but I might be biased! $\endgroup$ – Dikran Marsupial Feb 14 '15 at 13:03
  • 1
    $\begingroup$ @DikranMarsupial did you publish or do you have anything pre-publication available? $\endgroup$ – Tom Hale Nov 2 '17 at 5:05
0
$\begingroup$

The ELM "learns" from the data by analytically solving for the output weights. Thus the larger the data that is fed into the network will produce better results. However this also requires more numbers of hidden nodes. If the ELM is trained with little or no error, when given a new set of input, it is unable to produce the correct output.

The main advantage of ELM over traditional neural net such a back propagation is its fast training time. Most of the computation time is spent on solving the output layer weight as mentioned in Huang paper.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.