5
$\begingroup$

Often I have found deep learning and extreme learning machine discussed together.

Based on my little knowledge of the subject my impression is that they are different methods with different aims.

So what's the relation between deep learning and extreme learning machine ? Is there any important paper that I am missing and that I should read?

$\endgroup$
  • 9
    $\begingroup$ You usually have to wear a helmet to conduct extreme machine learning. $\endgroup$ – Sycorax Aug 26 '14 at 15:53
  • 1
    $\begingroup$ @user777 can you be clearer? $\endgroup$ – Donbeo Aug 26 '14 at 16:00
4
$\begingroup$

Extreme learning machines and deep learning are slightly related, but advocate quite adversary concepts.

ELMs are neural nets with a single hidden layer, where the first weight matrix is initialized randomly. This allows the output matrix to be estimated via least squares, which is very quickly done.

Deep learning, on the other hand, is the learning of deep architectures (e.g. deep neural nets). Depending on the strategy, all the layers are optimized jointly or greedily.

Long story short. ELM says: "only learn the last layer". Deep learning says: "Learn all the layers." It seems that DL is much more successfull than ELMs.

$\endgroup$
  • 1
    $\begingroup$ but my impression is that they have also different aim. ELM is like an intelligent way to train a simple neural network while DL is a way to define a new set of input variables. Do you agree? $\endgroup$ – Donbeo Aug 26 '14 at 21:03
  • 4
    $\begingroup$ No, I don't. ELM is a fast way to train shallow RNNs. Deep learning is more of a whole field, and its main characteristic is not the way of training (there are many methods) nor the task (supervised, semi-supervised, transfer, unsupervised, reinforcement learning--it has all been done). It's the kind of architectures used. $\endgroup$ – bayerj Aug 27 '14 at 13:20
8
$\begingroup$

The difference is: deep learning is original, while ELM is just a fancy name for 3 old methods.

The “extreme learning machines (ELM)” are indeed worth working on, but they just shouldn’t be called “ELM”. With annotated PDF files at http://elmorigin.wix.com/originofelm , you can easily verify the following facts within 10 to 20 minutes:

  1. The kernel (or constrained-optimization-based) version of ELM (ELM-Kernel, Huang 2012) is identical to kernel ridge regression (for regression and single-output classification, Saunders ICML 1998, as well as the LS-SVM with zero bias; for multiclass multi-output classification, An CVPR 2007).

  2. ELM-SLFN (the single-layer feedforward network version of the ELM, Huang IJCNN 2004) is identical to the randomized neural network (RNN, with omission of bias, Schmidt 1992) and another simultaneous work, i.e., the random vector functional link (RVFL, with omission of direct input-output links, Pao 1994).

  3. ELM-RBF (Huang ICARCV 2004) is identical to the randomized RBF neural network (Broomhead-Lowe 1988, with a performance-degrading randomization of RBF radii or impact factors).

  4. In all three cases above, G.-B. Huang got his papers published after excluding a large volume of very closely related literature.

  5. Hence, all 3 "ELM variants" have absolutely no technical originality, promote unethical research practices among researchers, and steal citations from original inventors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.