I wanted to find a variant of the perceptron which works for non-separable data, so I tried using $f(x)=\mathrm{\tanh}(x)$ instead of the hard threshold function and finding a $w$ that minimises the function $$\frac{1}{2} \sum_i (y_i - f(w \cdot x_i))^2$$ where $y_i \in \{-1,1\}$ are the classes of the training examples and $x_i$ are the features. I minimised it by using "batch" gradient descent and it seems to work nicely and gives me what I want.

My question is: is this the same as Adaline? I found a few references to Adaline, but I can't tell whether Adaline refers to the classifer itself, or also the algorithm which is used to train it, or even the physical machine which originally implemented that algorithm (by the way, I am equally confused about the use of "perceptron"). Am I justified in saying that I am using Adaline?


The name ADALINE (ADaptive LInear NEuron) come from both the physical implementation of an early classifier, but it is also the name specific design.

See: http://en.wikipedia.org/wiki/ADALINE

Apparently McCulloch-Pitts perceptrons came first. ADALINE was a variation on this that used a linear response function, as opposed to heaviside step. ADALINE is fitted using gradient decent because its output is the dot product of the weights and the inputs, effectively with a linear transfer function. The original McCulloch-Pitts neuron had a heaviside step transfer function, and so couldn't use gradient descent (I had this the wrong way round in my original answer). More general artificial neurons apply any transfer function and so, if differentiable, can be fitted with gradient descent.

As to being "equally confused about the use of perceptron" then really a perceptron is just a linear classifier. The main difference between a perceptron and other classifiers like logistic regression etc. is that they are "trained" using online algorithms - that is you can give them one datapoint at a time. Each input/response pair that you give it updates the weights and should make it a better classifier. In the early days the connection between perceptrons and logistic regression (and other classifiers) was not clear, but these days it is understood that they do the same thing, and you can "train" logistic regression one point at a time if you wish. Typically perceptrons are now discussed as the elements of larger neural networks or multilayer perceptrons. Some sources suggest that a perceptron must have a binary output, but then other sources on multilayer perceptrons don't enforce this.

For safety I would suggest you refered to your model as an artificial neuron. It isn't ADALINE, and it's not an original McCulloch-Pitts; it might be a perceptron, but it is probably best to refer to it as an artificial neuron.

Incidentally, Information Theory, Inference and Learning Algorithms by D. MacKay has almost exactly your case as an example (chapter 39) and refers to it simply as a "Single Neuron".

  • $\begingroup$ The learning rule given on the Wikipedia page for the ADALINE is gradient descent. $\endgroup$ – alto Feb 11 '13 at 18:56
  • $\begingroup$ Does any classifier with a "threshold function" and a "learning rule" count as a "McCulloch-Pitts neuron"? $\endgroup$ – Flounderer Feb 12 '13 at 3:00
  • $\begingroup$ @alto yes you are quite right, my memory failed me. My answer is clearly incorrect as it stands - I will edit to correct. That will teach me to read a link before I post it! $\endgroup$ – Corone Feb 12 '13 at 14:12
  • 1
    $\begingroup$ @Flounderer, tried to address that in corrected answer, but probably not. But McCulloch-Pitts was the first back in the 40s, and so all neurons inherit from those early works. I doubt you will get extra gravitas by giving it a grand name though. And if you get mixed up you look a bit of an idiot (like my first answer really...) $\endgroup$ – Corone Feb 12 '13 at 14:35
  • $\begingroup$ Both answers were useful, but I have decided to accept this one. The classifier is not ADALINE; ADALINE uses the identity activation function. $\endgroup$ – Flounderer Feb 15 '13 at 1:45

What you describe is essentially just logistic regression with a scaled output using squared loss rather than the usual log loss. Notice that $\tanh(x) = 2\sigma(x) - 1$ where

$$ \sigma(x) = \frac{1}{1 + e^{-x}} $$

is the logistic function. The decision boundary will still be linear.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.