Return to Answer

In short, logistic regression has probabilistic connotations that go beyond the classifier use in ML. I have some notes on logistic regression here.

The hypothesis in logistic regression provides a measure of uncertainty in the occurrence of a binary outcome based on a linear model. The output is bounded asymptotically between $0$ and $1$, and depends on a linear model, such that when the underlying regression line has value $0$, the logistic equation is $0.5 = \frac{e^0}{1+e^0}$, providing a natural cutoff point for classification purposes. However, it is at the cost of throwing out the probability information in the actual result of $h(\Theta^T\bf x) =\frac{e^{\Theta^T \bf x}}{1 +e^{\Theta^T\bf x}}$, which often is interesting (e.g. probability of loan default given income, credit score, age, etc.).

The perceptron classification algorithm is a more basic procedure, based on dot products between examples and weights. Whenever an example is misclassified the sign of the dot product is at odds with the classification value ($-1$ and $1$) in the training set. To correct this, the example vector will be iteratively added or subtracted from the vector of weights or coefficients, progressively updating its elements:

Vectorially, the $d$ features or attributes of an example are $\bf x$, and the idea is to "pass" the example if:

$\displaystyle \sum_{1}^d \theta_i x_i > \text{theshold}$ or...

$h(x) = \text{sign}\big(\displaystyle \sum_{1}^d \theta_i x_i - \text{theshold}\big)$. The sign function results in $1$ or $-1$, as opposed to $0$ and $1$ in logistic regression.

The threshold will be absorbed into the bias coefficient, $+ \theta_0$. The formula is now:

$h(x) = \text{sign}\big(\displaystyle \sum_0^d \theta_i x_i\big)$, or vectorized: $h(x) = \text{sign}(\theta^T\bf x)$.

Misclassified points will have $\text{sign}(\theta^T\bf x) \neq y_n$, meaning that the dot product of $\Theta$ and $\bf x_n$ will be positive (vectors in the same direction), when $y_n$ is negative, or the dot product will be negative (vectors in opposite directions), while $y_n$ is positive.

I have been working on the differences between these two methods in a dataset from the same course, in which the test results in two separate exams are related to the final acceptance to college:

The decision boundary can be easily found with logistic regression, but was interesting to see that although the coefficients obtained with perceptron were vastly different than in logistic regression, the simple application of the $\text{sign}(\cdot)$ function to the results yielded just as good a classifying algorithm. In fact the maximum accuracy (the limit set by linear inseparability of some examples) was reached by the second iteration. Here is the sequence of boundary division lines as $10$ iterations approximated the weights, starting from a random vector of coefficients:

The accuracy in the classification as a function of the number of iterations increases rapidly and plateaus at $90\%$, consistent how fast a near-optimal decision boundary is reached in the videoclip above. Here is the plot of the learning curve:

The code used is here.

In short, logistic regression has probabilistic connotations that go beyond the classifier use in ML. I have some notes on logistic regression here.

The hypothesis in logistic regression provides a measure of uncertainty in the occurrence of a binary outcome based on a linear model. The output is bounded asymptotically between $0$ and $1$, and depends on a linear model, such that when the underlying regression line has value $0$, the logistic equation is $0.5 = \frac{e^0}{1+e^0}$, providing a natural cutoff point for classification purposes. However, it is at the cost of throwing out the probability information in the actual result of $h(\Theta^T\bf x) =\frac{e^{\Theta^T \bf x}}{1 +e^{\Theta^T\bf x}}$, which often is interesting (e.g. probability of loan default given income, credit score, age, etc.).

The perceptron classification algorithm is a more basic procedure, based on dot products between examples and weights. Whenever an example is misclassified the sign of the dot product is at odds with the classification value ($-1$ and $1$) in the training set. To correct this, the example vector will be iteratively added or subtracted from the vector of weights or coefficients, progressively updating its elements:

Vectorially, the $d$ features or attributes of an example are $\bf x$, and the idea is to "pass" the example if:

$\displaystyle \sum_{1}^d \theta_i x_i > \text{theshold}$ or...

$h(x) = \text{sign}\big(\displaystyle \sum_{1}^d \theta_i x_i - \text{theshold}\big)$. The sign function results in $1$ or $-1$, as opposed to $0$ and $1$ in logistic regression.

The threshold will be absorbed into the bias coefficient, $+ \theta_0$. The formula is now:

$h(x) = \text{sign}\big(\displaystyle \sum_0^d \theta_i x_i\big)$, or vectorized: $h(x) = \text{sign}(\theta^T\bf x)$.

Misclassified points will have $\text{sign}(\theta^T\bf x) \neq y_n$, meaning that the dot product of $\Theta$ and $\bf x_n$ will be positive (vectors in the same direction), when $y_n$ is negative, or the dot product will be negative (vectors in opposite directions), while $y_n$ is positive.

I have been working on the differences between these two methods in a dataset from the same course, in which the test results in two separate exams are related to the final acceptance to college:

The decision boundary can be easily found with logistic regression, but was interesting to see that although the coefficients obtained with perceptron were vastly different than in logistic regression, the simple application of the $\text{sign}(\cdot)$ function to the results yielded just as good a classifying algorithm. In fact the maximum accuracy (the limit set by linear inseparability of some examples) was reached by the second iteration. Here is the sequence of boundary division lines as $10$ iterations approximated the weights, starting from a random vector of coefficients:

The accuracy in the classification as a function of the number of iterations increases rapidly and plateaus at $90\%$, consistent how fast a near-optimal decision boundary is reached in the videoclip above. Here is the plot of the learning curve:

The code used is here.

In short, logistic regression has probabilistic connotations that go beyond the classifier use in ML. I have some notes on logistic regression here.

The hypothesis in logistic regression provides a measure of uncertainty in the occurrence of a binary outcome based on a linear model. The output is bounded asymptotically between $0$ and $1$, and depends on a linear model, such that when the underlying regression line has value $0$, the logistic equation is $0.5 = \frac{e^0}{1+e^0}$, providing a natural cutoff point for classification purposes. However, it is at the cost of throwing out the probability information in the actual result of $h(\Theta^T\bf x) =\frac{e^{\Theta^T \bf x}}{1 +e^{\Theta^T\bf x}}$, which often is interesting (e.g. probability of loan default given income, credit score, age, etc.).

The perceptron classification algorithm is a more basic procedure, based on dot products between examples and weights. Whenever an example is misclassified the sign of the dot product is at odds with the classification value ($-1$ and $1$) in the training set. To correct this, the example vector will be iteratively added or subtracted from the vector of weights or coefficients, progressively updating its elements:

Vectorially, the $d$ features or attributes of an example are $\bf x$, and the idea is to "pass" the example if:

$\displaystyle \sum_{1}^d \theta_i x_i > \text{theshold}$ or...

$h(x) = \text{sign}\big(\displaystyle \sum_{1}^d \theta_i x_i - \text{theshold}\big)$. The sign function results in $1$ or $-1$, as opposed to $0$ and $1$ in logistic regression.

The threshold will be absorbed into the bias coefficient, $+ \theta_0$. The formula is now:

$h(x) = \text{sign}\big(\displaystyle \sum_0^d \theta_i x_i\big)$, or vectorized: $h(x) = \text{sign}(\theta^T\bf x)$.

Misclassified points will have $\text{sign}(\theta^T\bf x) \neq y_n$, meaning that the dot product of $\Theta$ and $\bf x_n$ will be positive (vectors in the same direction), when $y_n$ is negative, or the dot product will be negative (vectors in opposite directions), while $y_n$ is positive.

I have been working on the differences between these two methods in a dataset from the same course, in which the test results in two separate exams are related to the final acceptance to college:

The decision boundary can be easily found with logistic regression, but was interesting to see that although the coefficients obtained with perceptron were vastly different than in logistic regression, the simple application of the $\text{sign}(\cdot)$ function to the results yielded just as good a classifying algorithm. In fact the maximum accuracy (the limit set by linear inseparability of some examples) was reached by the second iteration. Here is the sequence of boundary division lines as $10$ iterations approximated the weights, starting from a random vector of coefficients:

The accuracy in the classification as a function of the number of iterations increases rapidly and plateaus at $90\%$, consistent how fast a near-optimal decision boundary is reached in the videoclip above. Here is the plot of the learning curve:

The code used is here.

In short, logistic regression has probabilistic connotations that go beyond the classifier use in ML. I have some notes on logistic regression here.

The hypothesis in logistic regression provides a measure of uncertainty in the occurrence of a binary outcome based on a linear model. The output is bounded asymptotically between $0$ and $1$, and depends on a linear model, such that when the underlying regression line has value $0$, the logistic equation is $0.5 = \frac{e^0}{1+e^0}$, providing a natural cutoff point for classification purposes. However, it is at the cost of throwing out the probability information in the actual result of $h(\Theta^T\bf x) =\frac{e^{\Theta^T \bf x}}{1 +e^{\Theta^T\bf x}}$, which often is interesting (e.g. probability of loan default given income, credit score, age, etc.).

The perceptron classification algorithm is a more basic procedure, based on dot products between examples and weights. Whenever an example is misclassified the sign of the dot product is at odds with the classification value ($-1$ and $1$) in the training set. To correct this, the example vector will be iteratively added or subtracted from the vector of weights or coefficients, progressively updating its elements:

Vectorially, the $d$ features or attributes of an example are $\bf x$, and the idea is to "pass" the example if:

$\displaystyle \sum_{1}^d \theta_i x_i > \text{theshold}$ or...

$h(x) = \text{sign}\big(\displaystyle \sum_{1}^d \theta_i x_i - \text{theshold}\big)$. The sign function results in $1$ or $-1$, as opposed to $0$ and $1$ in logistic regression.

The threshold will be absorbed into the bias coefficient, $+ \theta_0$. The formula is now:

$h(x) = \text{sign}\big(\displaystyle \sum_0^d \theta_i x_i\big)$, or vectorized: $h(x) = \text{sign}(\theta^T\bf x)$.

Misclassified points will have $\text{sign}(\theta^T\bf x) \neq y_n$, meaning that the dot product of $\Theta$ and $\bf x_n$ will be positive (vectors in the same direction), when $y_n$ is negative, or the dot product will be negative (vectors in opposite directions), while $y_n$ is positive.

I have been working on the differences between these two methods in a dataset from the same course, in which the test results in two separate exams are related to the final acceptance to college:

The decision boundary can be easily found with logistic regression, but was interesting to see that although the coefficients obtained with perceptron were vastly different than in logistic regression, the simple application of the $\text{sign}(\cdot)$ function to the results yielded just as good a classifying algorithm. In fact the maximum accuracy (the limit set by linear inseparability of some examples) was reached by the second iteration. Here is the sequence of boundary division lines as $10$ iterations approximated the weights, starting from a random vector of coefficients:

The accuracy in the classification as a function of the number of iterations increases rapidly and plateaus at $90\%$, consistent how fast a near-optimal decision boundary is reached in the videoclip above. Here is the plot of the learning curve:

The code used is here.

In short, logistic regression has probabilistic connotations that go beyond the classifier use in ML. I have some notes on logistic regression here.

The hypothesis in logistic regression provides a measure of uncertainty in the occurrence of a binary outcome based on a linear model. The output is bounded asymptotically between $0$ and $1$, and depends on a linear model, such that when the underlying regression line has value $0$, the logistic equation is $0.5 = \frac{e^0}{1+e^0}$, providing a natural cutoff point for classification purposes. However, it is at the cost of throwing out the probability information in the actual result of $h(\Theta^T\bf x) =\frac{e^{\Theta^T \bf x}}{1 +e^{\Theta^T\bf x}}$, which often is interesting (e.g. probability of loan default given income, credit score, age, etc.).

The perceptron classification algorithm is a more basic procedure, based on dot products between examples and weights. Whenever an example is misclassified the sign of the dot product is at odds with the classification value ($-1$ and $1$) in the training set. To correct this, the example vector will be iteratively added or subtracted from the vector of weights or coefficients, progressively updating its elements:

Vectorially, the $d$ features or attributes of an example are $\bf x$, and the idea is to "pass" the example if:

$\displaystyle \sum_{1}^d \theta_i x_i > \text{theshold}$ or...

$h(x) = \text{sign}\big(\displaystyle \sum_{1}^d \theta_i x_i - \text{theshold}\big)$. The sign function results in $1$ or $-1$, as opposed to $0$ and $1$ in logistic regression.

The threshold will be absorbed into the bias coefficient, $+ \theta_0$. The formula is now:

$h(x) = \text{sign}\big(\displaystyle \sum_0^d \theta_i x_i\big)$, or vectorized: $h(x) = \text{sign}(\theta^T\bf x)$.

Misclassified points will have $\text{sign}(\theta^T\bf x) \neq y_n$, meaning that the dot product of $\Theta$ and $\bf x_n$ will be positive (vectors in the same direction), when $y_n$ is negative, or the dot product will be negative (vectors in opposite directions), while $y_n$ is positive.

I have been working on the differences between these two methods in a dataset from the same course, in which the test results in two separate exams are related to the final acceptance to college:

The decision boundary can be easily found with logistic regression, but was interesting to see that although the coefficients obtained with perceptron were vastly different than in logistic regression, the simple application of the $\text{sign}(\cdot)$ function to the results yielded just as good a classifying algorithm. In fact the maximum accuracy (the limit set by linear inseparability of some examples) was reached by the second iteration. Here is the sequence of boundary division lines as $10$ iterations approximated the weights, starting from a random vector of coefficients:

The accuracy in the classification as a function of the number of iterations increases rapidly and plateaus at $90\%$, consistent how fast a near-optimal decision boundary is reached in the videoclip above. Here is the plot of the learning curve:

The code used is here.

In short, logistic regression has probabilistic connotations that go beyond the classifier use in ML. I have some notes on logistic regression here.

The hypothesis in logistic regression provides a measure of uncertainty in the occurrence of a binary outcome based on a linear model. The output is bounded asymptotically between $0$ and $1$, and depends on a linear model, such that when the underlying regression line has value $0$, the logistic equation is $0.5 = \frac{e^0}{1+e^0}$, providing a natural cutoff point for classification purposes. However, it is at the cost of throwing out the probability information in the actual result of $h(\Theta^T\bf x) =\frac{e^{\Theta^T \bf x}}{1 +e^{\Theta^T\bf x}}$, which often is interesting (e.g. probability of loan default given income, credit score, age, etc.).

The perceptron classification algorithm is a more basic procedure, based on dot products between examples and weights. Whenever an example is misclassified the sign of the dot product is at odds with the classification value ($-1$ and $1$) in the training set. To correct this, the example vector will be iteratively added or subtracted from the vector of weights or coefficients, progressively updating its elements:

Vectorially, the $d$ features or attributes of an example are $\bf x$, and the idea is to "pass" the example if:

$\displaystyle \sum_{1}^d \theta_i x_i > \text{theshold}$ or...

$h(x) = \text{sign}\big(\displaystyle \sum_{1}^d \theta_i x_i - \text{theshold}\big)$. The sign function results in $1$ or $-1$, as opposed to $0$ and $1$ in logistic regression.

The threshold will be absorbed into the bias coefficient, $+ \theta_0$. The formula is now:

$h(x) = \text{sign}\big(\displaystyle \sum_0^d \theta_i x_i\big)$, or vectorized: $h(x) = \text{sign}(\theta^T\bf x)$.

Misclassified points will have $\text{sign}(\theta^T\bf x) \neq y_n$, meaning that the dot product of $\Theta$ and $\bf x_n$ will be positive (vectors in the same direction), when $y_n$ is negative, or the dot product will be negative (vectors in opposite directions), while $y_n$ is positive.

I have been working on the differences between these two methods in a dataset from the same course, in which the test results in two separate exams are related to the final acceptance to college:

The decision boundary can be easily found with logistic regression, but was interesting to see that although the coefficients obtained with perceptron were vastly different than in logistic regression, the simple application of the $\text{sign}(\cdot)$ function to the results yielded just as good a classifying algorithm. In fact the maximum accuracy (the limit set by linear inseparability of some examples) was reached by the second iteration. Here is the sequence of boundary division lines as $10$ iterations approximated the weights, starting from a random vector of coefficients:

The accuracy in the classification as a function of the number of iterations increases rapidly and plateaus at $90\%$, consistent how fast a near-optimal decision boundary is reached in the videoclip above. Here is the plot of the learning curve:

The code used is here.