What is the name of this perceptron-like classifier?

Question

I wanted to find a variant of the perceptron which works for non-separable data, so I tried using $f(x)=\mathrm{\tanh}(x)$ instead of the hard threshold function and finding a $w$ that minimises the function $$\frac{1}{2} \sum_i (y_i - f(w \cdot x_i))^2$$ where $y_i \in \{-1,1\}$ are the classes of the training examples and $x_i$ are the features. I minimised it by using "batch" gradient descent and it seems to work nicely and gives me what I want.

My question is: is this the same as Adaline? I found a few references to Adaline, but I can't tell whether Adaline refers to the classifer itself, or also the algorithm which is used to train it, or even the physical machine which originally implemented that algorithm (by the way, I am equally confused about the use of "perceptron"). Am I justified in saying that I am using Adaline?

Answer 1

The name ADALINE (ADaptive LInear NEuron) come from both the physical implementation of an early classifier, but it is also the name specific design.

See: http://en.wikipedia.org/wiki/ADALINE

Apparently McCulloch-Pitts perceptrons came first. ADALINE was a variation on this that used a linear response function, as opposed to heaviside step. ADALINE is fitted using gradient decent because its output is the dot product of the weights and the inputs, effectively with a linear transfer function. The original McCulloch-Pitts neuron had a heaviside step transfer function, and so couldn't use gradient descent (I had this the wrong way round in my original answer). More general artificial neurons apply any transfer function and so, if differentiable, can be fitted with gradient descent.

As to being "equally confused about the use of perceptron" then really a perceptron is just a linear classifier. The main difference between a perceptron and other classifiers like logistic regression etc. is that they are "trained" using online algorithms - that is you can give them one datapoint at a time. Each input/response pair that you give it updates the weights and should make it a better classifier. In the early days the connection between perceptrons and logistic regression (and other classifiers) was not clear, but these days it is understood that they do the same thing, and you can "train" logistic regression one point at a time if you wish. Typically perceptrons are now discussed as the elements of larger neural networks or multilayer perceptrons. Some sources suggest that a perceptron must have a binary output, but then other sources on multilayer perceptrons don't enforce this.

For safety I would suggest you refered to your model as an artificial neuron. It isn't ADALINE, and it's not an original McCulloch-Pitts; it might be a perceptron, but it is probably best to refer to it as an artificial neuron.

Incidentally, Information Theory, Inference and Learning Algorithms by D. MacKay has almost exactly your case as an example (chapter 39) and refers to it simply as a "Single Neuron".

Answer 2

What you describe is essentially just logistic regression with a scaled output using squared loss rather than the usual log loss. Notice that $\tanh(x) = 2\sigma(x) - 1$ where

$$ \sigma(x) = \frac{1}{1 + e^{-x}} $$

is the logistic function. The decision boundary will still be linear.