9
$\begingroup$

Recently, a project I'm involved in made use of a linear perceptron for multiple (21 predictor) regression. It used stochastic GD. How is this different from OLS linear regression?

$\endgroup$
  • $\begingroup$ The Perceptron class you link to is for a classifier (binary output) rather than a regressor (continuous output). Is that the actual code you used? If so, that's the difference. :) $\endgroup$ – djs Mar 30 '15 at 3:03
  • $\begingroup$ @Dougal, it still counts among the GLMs though: scikit-learn.org/stable/… $\endgroup$ – Simon Kuang Mar 30 '15 at 3:30
  • $\begingroup$ @Dougal: suppose you had a (G)LM that you optimized to L2 using SGDRegressor; would this be equivalent to linear regression? $\endgroup$ – Simon Kuang Mar 30 '15 at 3:34
  • $\begingroup$ Yes, some GLMs are classifiers. If you used SGDRegressor(loss='squared_loss', penalty='none'), that is OLS. $\endgroup$ – djs Mar 30 '15 at 3:57
5
$\begingroup$

scikit-learn's Perceptron class (equivalent to SGDClassifier(loss="perceptron", penalty=None, learning_rate="constant", eta0=1)) uses the following objective function: $$\frac{1}{N} \sum_{i=1}^N \max(0, - y_i w^T x_i).$$ In this case, $y_i \in \{-1, 1\}$. If $w^T x_i$ has the right sign, it doesn't incur any loss; otherwise, it gives linear loss. The perceptron in particular uses a fixed learning rate which can lead to some optimization weirdness as well.

Least squares regression, by contrast, uses $$\frac{1}{N} \sum_{i=1}^N (y_i - w^T x_i)^2.$$ Here $y_i$ can be any real; you can give it classification targets in $\{-1, 1\}$ if you want, but it's not going to give you a very good model. You can optimize this with SGDRegressor(loss="squared_loss", penalty=None) if you'd like.

The two define fundamentally different models: the perceptron predicts a binary class label with $\mathrm{sign}(w^T x_i)$, whereas linear regression predicts a real value with $w^T x_i$. This answer talks some about why trying to solve a classification problem with a regression algorithm can be problematic.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.