1
$\begingroup$

Suppose we have a training dataset with $s$ instances, and we split it to 2 sets, of lengths $n, m$ such that $s = n + m$ . We train two perceptron models $p$ and $q$, both are defined by a k-dimensional vector of weights, $w_1$ and $w_2$. Such that $p$ was trained on $n$ instances, and $q$ on $m$ instances. In order to combine them to a single model, is it possible to simply use a weighted average of the weights? i.e.:

$w_3 = (1/(n+m))[n * w_1 + m * w_2]$

$\endgroup$
1
$\begingroup$

For deciding how to combine the results of two perceptrons, it might pay to consider what are the convergence guarantees of a single perceptron (at least in the classical version).

Given a linearly separable set, a perceptron is only guaranteed to output some separating hyperplane. E.g., each of the two hyperplanes in the following diagram (taken from the Wikipedia entry) is valid:

enter image description here

In particular, a perceptron is not guaranteed to find the hyperplane which is equi-distanced to the points it separates - its convergence makes no such considerations (as opposed to the result of an SVM, for example).

FWIW, based on this, I don't think your proposed solution displays some properties that a combination of SVMs should intuitively have. For example, logically, if $n \gg m$, we would expect that effectively only the first SVM should affect the results of the combined SVM. This is not necessarily the case.

Consider the following dataset consisting of four equi-sized clusters of points.

enter image description here

with the red points indicating 0, and the black points indicating 1.

Say you train the second perceptron on a small number of points $m$. The chance that all these points are taken from the extreme groups $A$ and $D$ is $\left( \frac{1}{2} \right)^m$ independent from the size of the sets or their distance from each other. If this is the case, note that the separating hyperplane could be anywhere between A and D. Playing around with the distances of the clusters, this could affect every combined classification, even with the weighted scheme you suggest.

Note that for high-dimension problems, there are several chances of this type of problem occurring.

Because of this, I'd suggest conservatively combining two perceptrons as simply the output of the perceptron trained on the majority of points - it at least avoids this problem.

$\endgroup$
  • $\begingroup$ Right, but why is it "independent from the size of the sets"? I thought the 1/2 comes from A and D having 8 instances out of 16 total. Or did you mean by "size" the max distance between a pair of instances within each set? $\endgroup$ – dimid Oct 18 '16 at 10:03
  • $\begingroup$ What I meant was that if $A$, $B$, $C$, and $D$, had each 100 or 1,000,000 elements, the probability of choosing $A$ and $D$ only would only depend on $m$. I really liked your question, BTW - caused me to think on this problem. $\endgroup$ – Ami Tavory Oct 18 '16 at 10:34
  • $\begingroup$ Thanks, BTW I saw in your profile that you're learning Haskell. My final goal is to adapt the perceptron or one of its variations (e.g. voted, averaged) to obey the monoid laws over the set of models (i.e. weights). $\endgroup$ – dimid Oct 18 '16 at 12:05
  • $\begingroup$ @dimid Oh, very cool and interdisciplinary goal! $\endgroup$ – Ami Tavory Oct 18 '16 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.