Questions tagged [perceptron]
An early example of neural network without any hidden layers and with a single (possibly nonlinear) output unit.
167 questions
45
votes
6
answers
58k
views
What's the difference between logistic regression and perceptron?
I'm going through Andrew Ng's lecture notes on Machine Learning.
The notes introduce us to logistic regression and then to perceptron. While describing Perceptron, the notes say that we just change ...
- 961
30
votes
3
answers
43k
views
Multi-layer perceptron vs deep neural network
This is a question of terminology. Sometimes I see people refer to deep neural networks as "multi-layered perceptrons", why is this? A perceptron, I was taught, is a single layer classifier (or ...
- 1,275
22
votes
3
answers
16k
views
From the Perceptron rule to Gradient Descent: How are Perceptrons with a sigmoid activation function different from Logistic Regression?
Essentially, my question is that in multilayer Perceptrons, perceptrons are used with a sigmoid activation function. So that in the update rule $\hat{y}$ is calculated as
$$\hat{y} = \frac{1}{1+\exp(...
user39663
20
votes
1
answer
6k
views
Why pure exponent is not used as activation function for neural networks?
The ReLU function is commonly used as an activation function in machine learning, as well, as its modifications (ELU, leaky ReLU).
The overall idea of these functions is the same: before ...
- 303
16
votes
4
answers
28k
views
What is the difference between a neural network and a perceptron?
Is there any difference between the terms "neural network" and "perceptron"?
- 13.1k
16
votes
1
answer
22k
views
Clarification about Perceptron Rule vs. Gradient Descent vs. Stochastic Gradient Descent implementation
I experimented a little bit with different Perceptron implementations and want to make sure if I understand the "iterations" correctly.
Rosenblatt's original perceptron rule
As far as I understand, ...
user39663
15
votes
3
answers
51k
views
Decision boundary plot for a perceptron
I am trying to plot the decision boundary of a perceptron algorithm and I am really confused about a few things. My input instances are in the form $[(x_{1},x_{2}), y]$, basically a 2D input instance (...
- 313
11
votes
4
answers
21k
views
Difference between MLP(Multi-layer Perceptron) and Neural Networks?
I am wondering about the differences. Based on my understanding, MLP is one kind of neural networks, where the activation function is sigmoid, and error term is cross-entropy(logistics) error. Looking ...
- 547
11
votes
1
answer
6k
views
How to kernelize a simple perceptron?
Classification problems with nonlinear boundaries cannot be solved by a simple perceptron. The following R code is for illustrative purposes and is based on this example in Python):
...
- 6,246
10
votes
4
answers
17k
views
Can a perceptron with sigmoid activation function perform nonlinear classification?
Consider the perceptron as illustrated in the figure above.
I know:
If the activation function is linear, i.e. the first three cases, then
the perceptron is equivalent to a linear classifier.
...
- 243
9
votes
1
answer
704
views
Support Vector Machine with Perceptron Loss
Typical support vector classifier uses the following optimization procedure:
$$\min \dfrac{1}{2}||w||^2 + C\sum_{i=1}^N \zeta_i$$
$$y_i(w^Tx_i+b) \geq 1 - \zeta_i$$
$$\zeta_i \geq 0$$
This hinge loss ...
- 4,504
8
votes
2
answers
3k
views
Intuition behind perceptron algorithm with offset
I was looking for an intuition for the perceptron algorithm with offset rule, why the update rule is as follows:
cycle through all points until convergence:
$\textbf{if }\, y^{(t)} \neq \theta^{T}x^{(...
- 7,074
8
votes
1
answer
2k
views
Why aren't neural networks used with RBF activation functions (or other non-monotonic ones)?
In most work I've seen, MLPs (multilayer perceptron, the most typical feedforward neural network) and RBF (radial basis function) networks are compared as distinct models, where
MLP neuron outputs $\...
7
votes
2
answers
24k
views
What is the difference between MLP and RBF?
What are the main differences between two types of feedforward networks such as multilayer perceptrons (MLP) and radial basis function (RBF)?
What are the fundamental differences between these two ...
- 559
7
votes
1
answer
10k
views
When to use RBF networks instead of multilayer perceptron?
I understand that a radial basis function neural network (RBF) usually has 1 hidden layer, and it differs from a multi-layer perceptron (MLP) via its activation and combination functions among other ...
- 211
5
votes
2
answers
177
views
Confusion regarding the criteria for defining a ML model as a linear model
I am confused about the criteria which determines whether a model is linear or not. As far as I understand, the following statements are equivalent :
A model is linear
Output class label is a linear ...
- 53
5
votes
2
answers
2k
views
SciKit Learn: Multilayer perceptron early stopping, restore best weights
In the SciKit documentation of the MLP classifier, there is the early_stopping flag which allows to stop the learning if there is not any improvement in several ...
- 747
5
votes
2
answers
4k
views
How to make the conclustion that VC-dimension for hyperplane in $\mathbb{R^{3}}$ is strictly less than $\mathbb{R^{4}}$?
Given four points in $\mathbb{R^{3}}$ real space,
$S = \{(1, 1, 1), (1, 1, -1), (-1, -1, 1), (-1, -1, -1)\} \in \mathbb{R^{3}}$.
Will these four points be shattered by a hyperplane in $\mathbb{R^{3}}...
- 153
5
votes
2
answers
3k
views
Formula for decision boundary of a classifier (in order to visualize it)
I'm confused on how to plot decision boundary for classifiers.
For example, i'm working with perceptron. So, the formula for decision boundary(if I understand this correctly) is
...
- 161
5
votes
1
answer
794
views
Why do nodes in hidden layer produce different results?
Assuming a simple, fully connected Multilayer Perceptron network with one input layer, one hidden layer with multiple nodes and one output layer. In this case the nodes in hidden layer are ...
- 450
5
votes
1
answer
2k
views
Deriving the line for the decision boundary
Let's say I have the following perceptron activation line:
$y = 1$ if $\sum_{i}^{d} w_ix_i \geq \theta $ and $y=0$ else. Now, for d = 4, I have derived the following question.
$$ w_4x_4+w_3x_3+w_2x_2+...
- 1,932
5
votes
1
answer
1k
views
Machine learning: intuition behind perceptron learning algorithm
Given features $x_1...x_n$, weights $w_1...w_n$, calculated output $y = W^T \cdot X$, and actual output $\hat{y}$, the perceptron learning algorithm changes the weights after each iteration as follows:...
- 51
5
votes
2
answers
3k
views
Why perceptron is linear classifier?
It is said that perceptron is linear classifier, but it has a non-linear activation function f = 1 if wx - b >= 0 and f = 0 otherwise
If i will use some non-linear function on linear combination of ...
- 71
5
votes
0
answers
2k
views
Adding regularization term to perceptron weight update
I'm trying to reduce overfitting on my perceptron network by adding in a regularization term. However I am not sure where the actual term goes...
Usually the regularization term is shown in the ...
- 51
4
votes
2
answers
3k
views
Can non-linearly separable data always be made linearly separable?
A data set that is linearly separable is a precondition for algorithms like the perceptron to converge. It's well-known that we can project low-dimensional data to a higher dimension using kernel ...
- 141
4
votes
2
answers
2k
views
Visualizing High Dimensional weight space for perceptrons
I am watching the Neural Network videos by Prof. Geoff Hinton. In there he talks about a high dimensional Weight Space for perceptrons.
In particular, I am ...
- 647
4
votes
1
answer
172
views
How do I check if the weights of my perceptron/step activation function are correct
I am new to stack overflow and deep learning so I hope I am doing this the right way. I tried to find the solution myself but it has not been successful so I am seeking some help. This is the ...
- 43
4
votes
1
answer
925
views
Over which set of elements should I perform norm clipping of gradients for backpropagation?
I want to normalise the gradients of my multi-layer perceptron in order to avoid the Exploding Gradients Problem, so I thought I would use l2-normalisation but am unsure about how to apply it to the ...
- 384
4
votes
1
answer
1k
views
Neural Networks - Difference between one dimensional layer vs multi-dimensional layer
Please take a look at these Neural Network architectures:
I can understand the architecture of Hidden Layer in Net-2 : you add 12 Neurons to your hidden layer...
What I can't understand is the 2-...
- 515
4
votes
1
answer
87
views
Why there are two different versions for batch perceptron algorithm?
In the book "Understanding Machine Learning, S. David Ben et al.", the authors describe the Batch Perceptron Algorithm as follows:
However, in the book "Python Machine Learning, ...
- 670
4
votes
1
answer
4k
views
What is the correct way of calculating Rectifier Linear and MaxOut functions?
If I have a artificial neuron with 2 inputs:
input 1 = 0.7 & weight = 0.7
input 2 = 0.3 & weight = 0.3
If I use a Rectifier Linear (ReLU) as activation ...
- 1,151
4
votes
1
answer
4k
views
What Is the Loss (Objective) Function for Linear Discriminant Analysis (LDA)?
As many algorithms can be viewed as optimization problems through the Loss function, I was wondering if such a loss function existed for LDA (linear classification). And if yes, what would it be ?
I ...
- 385
4
votes
2
answers
180
views
Understanding Weight Updates in a Perceptron
I am learning about perceptrons and how they work. I read that each weight $w_j$ is updated based on the equation:
$\begin{equation}
w_j:=w_j+\Delta w_j
\end{equation} $
Where:
$\begin{equation}
\...
- 41
3
votes
1
answer
285
views
Does the n >> p holds also for minibatches
It's pretty known that when dealing with models (without regularization) the main assumption is $n >> p$ where $p$ is the number of features in the dataset
Let's suppose that we have 1.000.000 ...
- 1,381
3
votes
1
answer
7k
views
Derivation of Perceptron weight update formula
I've started out studying Machine Learning and am currently reading up about how a single perceptron works. From the wikipedia page, my understanding is as follows: suppose we have an input sample $\...
- 750
3
votes
1
answer
456
views
MLP - what did author have in mind?
I am currently going through Machine learning and pattern recognition and I have following dilemma:
A algorithm for MLP is presented in that way
and corresponding python code to equation 4.9 (which ...
- 143
3
votes
1
answer
129
views
About Multi-layer Perceptrons
I've always been a bit confused when it comes to Deep Learning terminology.
Is the definition of the perceptron, whether single layer or multi layer, associated with a specific type of activation ...
- 101
3
votes
1
answer
121
views
Best approach for energy demand forecasting
I am trying to predict the amount of energy demand(Wh) of the next two weeks per hour. The dataset I have, contains each hour of each day since 2019 of the energy demand, is something like this:
...
- 53
3
votes
1
answer
4k
views
Calculation of decision boundaries with Perceptron
here i will train perceptron and plot decision boundaries (target is generated so I am sure that it is lineary separable). For sake of the example, there is no bias.
...
- 683
3
votes
1
answer
1k
views
Convergence speed of perceptron algorithm
I was reading the convergence proof for the perceptron algorithm. It says under the assumption that there are some $R$, $\theta^*$ with $|\theta^*| = 1$ and $\gamma > 0$, such that $y_t(x_t\cdot \...
- 33
3
votes
1
answer
800
views
Can this network learn the XOR function?
Let's say I have the following constraints:
The architecture is fixed (see image) (note that there are no biases)
Activation function for the hidden layer is ReLU
There's no activation function for ...
- 39
3
votes
1
answer
3k
views
Function approximation using multilayer perceptron (neural network)
I've been asked to solve a problem for a project. I'm working on Python or R. I need to approximate a function with multiplayer perceptron (neural network).
The function is: $y= 2\text{cos}(x)+4$ on ...
- 31
3
votes
0
answers
56
views
What's the unified definition of Tikhonov regularization
I met with "Tikhonov regularization" in two textbooks. The first is "Pattern Recognition and Machine Learning" by Christopher M. Bishop. In page 267 of his book, the regularized ...
- 333
3
votes
2
answers
825
views
How to find multi-layer perceptron weights?
I want to use a multi-layer perceptron to design the following function :
The architecture I want to use is the following one :
What would be $w_i$ weights ? Is there any guide to find them ?
I ...
2
votes
1
answer
2k
views
What are the best R packages for a classification problem with use of Neural networks [closed]
Surfing on the internet shows me that there are a lot of different packages and functions which can be used to train neural networks via R.
packages such as 'RSNNS', 'nnet','neuralnet', etc.
I'm ...
- 135
2
votes
3
answers
3k
views
Can a Single Layer Perceptron Learn a Nonlinear Function?
A lot of times I've read that with a Single Layer Perceptron (SLP), you can only learn linear functions. But what if we use a generalized linear model?
Imagine we only have one input feature $x_1$. ...
- 31
2
votes
1
answer
917
views
What is the expression for derivative of the signum function one should use in the BP training method
The back propagation learning method requires knowing of derivatives of activation functions. But what expression one should use for signum activation function
$$
\mathrm{Signum}( x ) =
\...
- 183
2
votes
2
answers
5k
views
Perceptron overfitting?
I'm trying to judge the performance of my perceptron linear discriminant. In one instance I'm training on a sample size of 150 and on another I'm training on a sample size of 1500. I test both of ...
- 111
2
votes
1
answer
386
views
Can we use perceptron training algorithm to train a single neuron with sigmoid activation?
The perceptron training algorithm is summarized as:
Apply the inputs and calculate the output $ y $
Compare with the desired output yd and calculate error $e = y-y_d$
Update the weights based on the ...
- 119
2
votes
1
answer
1k
views
Why does Perceptron use L1 norm as its error function?
Usually, people use L2 norm as a machine learning error function.
per wiki
the error function for a Perceptron model is
${\displaystyle {\frac {1}{s}}\sum _{j=1}^{s}|d_{j}-y_{j}(t)|}$
why is that?
- 2,005