34
votes
Accepted
Why pure exponent is not used as activation function for neural networks?
I think the most prominent reason is stability. Think about having consequent layers with exponential activation, and what happens to the output when you input a small number to the NN (e.g. $x=1$), ...
- 58.2k
29
votes
Accepted
Multi-layer perceptron vs deep neural network
One can consider multi-layer perceptron (MLP) to be a subset of deep neural networks (DNN), but are often used interchangeably in literature.
The assumption that perceptrons are named based on their ...
- 858
14
votes
What's the difference between logistic regression and perceptron?
There is some confusion that may arise here. Originally a perceptron was only referring to neural networks with a step function as the transfer function. In that case of course the difference is that ...
- 2,671
12
votes
Can a perceptron with sigmoid activation function perform nonlinear classification?
The network in the diagram has an input layer and an output layer, but no hidden layers. This type of network can't perform nonlinear classification or implement arbitrary nonlinear functions, ...
- 33.2k
11
votes
Multi-layer perceptron vs deep neural network
Good question: note that in the field of Deep Learning things are not always as well-cut and clearly defined as in Statistical Learning (also because there's a lot of hype), so don't expect to find ...
- 18.4k
8
votes
Accepted
Does the n >> p holds also for minibatches
I believe that this question arises from a misunderstanding about how neural networks are estimated compared to regression models. Consider the example of an OLS regression. In the OLS regression, the ...
- 94.1k
7
votes
Accepted
Confusion regarding the criteria for defining a ML model as a linear model
In statistics, we call a model linear when the outcome is a linear combination of the parameters, meaning that you can write $\hat y_i = \sum_j x_{ij}\hat\beta_j$ (second definition). We could have ...
- 67.1k
6
votes
From the Perceptron rule to Gradient Descent: How are Perceptrons with a sigmoid activation function different from Logistic Regression?
Using gradient descent, we optimize (minimize) the cost function
$$J(\mathbf{w}) = \sum_{i} \frac{1}{2}(y_i - \hat{y_i})^2 \quad \quad y_i,\hat{y_i} \in \mathbb{R}$$
If you minimize the mean ...
- 16.8k
6
votes
Accepted
Support Vector Machine with Perceptron Loss
Maximizing the margin is not just "rhetoric". It is the essential feature of support vector machines and ensures that the trained classifier has the optimal generalization properties. More ...
- 9,678
4
votes
What's the difference between logistic regression and perceptron?
You can use logistic regression to build a perceptron. The logistic regression uses logistic function to build the output from a given inputs. Logistic function produces a smooth output between 0 and ...
- 62.3k
4
votes
What is weights in perceptron
Using this picture I found online, you can see that the perceptron creates $y$ which is just the weighted sum of the inputs multiplied by an activation function (in the image it's a step function). So ...
- 1,437
4
votes
Accepted
MLP - what did author have in mind?
In formula $4.9$:
$$\delta_h(\zeta) = \bar{g}(h_\zeta)\sum_{k=1}^N w_\zeta \delta_o(k),\text{ where }\bar{g}=g(h_\zeta)(1-g(h_\zeta)) ,$$
we have a $\zeta$-th neuron in a hidden layer which is ...
- 625
4
votes
What is the expression for derivative of the signum function one should use in the BP training method
The function you wrote is completely flat, except at $x=\gamma$ where the step occurs. So, its derivative w.r.t. $x$ is zero everywhere, except at $x=\gamma$, where the derivative doesn't exist. This ...
- 33.2k
4
votes
Accepted
Derivation of Perceptron weight update formula
The difference is that the first formula is the derivation of just the output of a perceptron, while the second is the derivation of the non-linear activation of the perceptron.
When stacking ...
- 753
4
votes
Accepted
Why is the equation for a single-neuron perceptron decision boundary Wp + b = 0 set to ZERO?
That's linear algebra, not specific to perceptrons or machine learning. Let's start with a simple, two-dimensional case, with axes $x_1$ and $x_2$. Your class boundary can be defined by a straight ...
- 9,678
4
votes
Can non-linearly separable data always be made linearly separable?
For a given, finite data set it should always be possible—just let each data point have its own dimension! So, maybe a more interesting question would be for a stochastic model, generating a ...
- 82.8k
4
votes
Accepted
How to draw the single perceptron decision boundary when weights and bias are 0?
TL;DR: Your decision boundary is the whole $(x_1, x_2)$ plane.
In detail: The function
$$
z = w_1 x_1 + w_2 x_2 - b
$$
is a plane in the 3D space, spanned by the axes $(x_1, x_2, z)$. Where $z = 0$, ...
- 9,678
3
votes
Accepted
Deriving the line for the decision boundary
The key observation is that the boundary is exactly
$ w_4x_4+w_3x_3+w_2x_2+w_1x_1 + w_o = 0$,
and that this is exactly the equation of a hyperplane. So the question is how to find the point on this ...
- 4,603
3
votes
Intuition behind perceptron algorithm with offset
Some geometric intuition: the bias term $\theta_0$ is related to the distance of the separating hyperplane from the origin. More precisely, the distance of the decision hyperplane $\theta^Tx+\theta_0=...
- 751
3
votes
Formula for decision boundary of a classifier (in order to visualize it)
For logistic regression, it's actually
\begin{gather}
p(x) = \frac{1}{1 + \exp(- (\sum_j W_j x_j + b) )} = \frac12
\\ 1 + \exp(- (\sum_j W_j x_j + b) ) = 2
\\ \exp(- (\sum_j W_j x_j + b) ) = 1
\\ - (\...
- 25.2k
3
votes
Accepted
Visualizing High Dimensional weight space for perceptrons
As you're likely aware, a vector in a $D$-dimensional space can be described by a direction and a magnitude. The direction requires $D-1$ values to describe, and the magnitude requires one value to ...
- 3,454
3
votes
Accepted
Why do nodes in hidden layer produce different results?
Generically, when the network learns different weights for each node, it does so because the fit is better.* The optimization procedure has the goal of reducing the error, so if configurations with ...
- 94.1k
3
votes
Accepted
Over which set of elements should I perform norm clipping of gradients for backpropagation?
What is the global norm?
It's just the norm over all gradients as if they were concatenated together to form one global vector.
So regarding that question, you have to compute ...
3
votes
Accepted
No need for bias term if data is standardised? Linear classification models
Generally, no. There is no guarantee that the separating hyperplane is passing through the data mean. Consider the following case: The data has been standardized, but you still need the bias term to ...
- 11.5k
3
votes
Accepted
Rule of thumb Overfit in a MLP or is it possible with N = 135
Is it possible to train a 3 layer multilayer perceptron (mlp) in a
binary classification problem with just 135 observations with out
massive over fitting?
Yes, it is.
At least 10 observations /...
- 19.5k
3
votes
Why does Perceptron use L1 norm as its error function?
Note that the perceptron is a binary classifier, so its output $d_j$ for each input will be in $\{0,1\}$, and it will be compared to the known value $y_j$ which is also in $\{0,1\}$. Convince yourself ...
- 1,411
3
votes
Can a Single Layer Perceptron Learn a Nonlinear Function?
Just to add onto Bryan Krause's answer, you seem to misunderstand what the linear in linear model refers to. A regression model is called linear if the model parameters only appear as linear terms. In ...
- 1,368
3
votes
Accepted
Equivalence of Deep Feed-Forward Neural Network to Single-Layer Network
This follows from the fact that matrix multiplications by matrix A and then by matrix B is equivalent to multiplication by matrix C = AB. Linear activations are matrix multiplications (by the weight ...
- 514
3
votes
Accepted
SciKit Learn: Multilayer perceptron early stopping, restore best weights
Yes.
Inspecting the source code on GitHub, we see that the internal function _update_no_improvement_count keeps track of the best coefficients in cases where the ...
- 9,923
3
votes
Can we use perceptron training algorithm to train a single neuron with sigmoid activation?
Assume you use a sigmoid neuron with binary cross entropy loss:
$$H(y)=y_d\log y+(1-y_d)\log(1-y)$$
where $y=\sigma(w^Tx)$. The gradient wrt $w$ would be:
$$\begin{align}\frac{\partial H}{\partial w}&...
- 58.2k
Only top scored, non community-wiki answers of a minimum length are eligible