Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [perceptron]

An early example of neural network without any hidden layers and with a single (possibly nonlinear) output unit.

26
votes
6answers
14k views

What's the difference between logistic regression and perceptron?

I'm going through Andrew Ng's lecture notes on Machine Learning. The notes introduce us to logistic regression and then to perceptron. While describing Perceptron, the notes say that we just change ...
21
votes
3answers
11k views

From the Perceptron rule to Gradient Descent: How are Perceptrons with a sigmoid activation function different from Logistic Regression?

Essentially, my question is that in multilayer Perceptrons, perceptrons are used with a sigmoid activation function. So that in the update rule $\hat{y}$ is calculated as $$\hat{y} = \frac{1}{1+\exp(...
15
votes
1answer
10k views

Clarification about Perceptron Rule vs. Gradient Descent vs. Stochastic Gradient Descent implementation

I experimented a little bit with different Perceptron implementations and want to make sure if I understand the "iterations" correctly. Rosenblatt's original perceptron rule As far as I understand, ...
13
votes
3answers
10k views

Multi-layer perceptron vs deep neural network

This is a question of terminology. Sometimes I see people refer to deep neural networks as "multi-layered perceptrons", why is this? A perceptron, I was taught, is a single layer classifier (or ...
12
votes
4answers
12k views

What is the difference between a neural network and a perceptron?

Is there any difference between the terms "neural network" and "perceptron"?
10
votes
2answers
22k views

Decision boundary plot for a perceptron

I am trying to plot the decision boundary of a perceptron algorithm and I am really confused about a few things. My input instances are in the form $[(x_{1},x_{2}), y]$, basically a 2D input instance (...
10
votes
1answer
2k views

How to kernelize a simple perceptron?

Classification problems with nonlinear boundaries cannot be solved by a simple perceptron. The following R code is for illustrative purposes and is based on this example in Python): ...
7
votes
1answer
6k views

When to use RBF networks instead of multilayer percetron?

I understand that a radial basis function neural network (RBF) usually has 1 hidden layer, and it differs from a multi-layer perceptron (MLP) via its activation and combination functions among other ...
7
votes
3answers
7k views

Difference between MLP(Multi-layer Perceptron) and Neural Networks?

I am wondering about the differences. Based on my understanding, MLP is one kind of neural networks, where the activation function is sigmoid, and error term is cross-entropy(logistics) error. Looking ...
5
votes
2answers
8k views

What is the difference between MLP and RBF?

What are the main differences between two types of feedforward networks such as multilayer perceptrons (MLP) and radial basis function (RBF)? What are the fundamental differences between these two ...
4
votes
2answers
1k views

Formula for decision boundary of a classifier (in order to visualize it)

I'm confused on how to plot decision boundary for classifiers. For example, i'm working with perceptron. So, the formula for decision boundary(if I understand this correctly) is ...
4
votes
2answers
2k views

How to make the conclustion that VC-dimension for hyperplane in $\mathbb{R^{3}}$ is strictly less than $\mathbb{R^{4}}$?

Given four points in $\mathbb{R^{3}}$ real space, $S = \{(1, 1, 1), (1, 1, -1), (-1, -1, 1), (-1, -1, -1)\} \in \mathbb{R^{3}}$. Will these four points be shattered by a hyperplane in $\mathbb{R^{3}}...
4
votes
0answers
769 views

Intuition behind perceptron algorithm with offset

I was looking for an intuition for the perceptron algorithm with offset rule, why the update rule is as follows: cycle through all points until convergence $\text{if }\, y^{(t)} \neq \theta^{T}x^{(t)...
3
votes
3answers
5k views

Can a perceptron with sigmoid activation function perform nonlinear classification?

Consider the perceptron as illustrated in the figure above. I know: If the activation function is linear, i.e. the first three cases, then the perceptron is equivalent to a linear classifier. ...
3
votes
1answer
280 views

MLP - what did author have in mind?

I am currently going through Machine learning and pattern recognition and I have following dilemma: A algorithm for MLP is presented in that way and corresponding python code to equation 4.9 (which ...
3
votes
1answer
624 views

Neural Networks - Difference between one dimensional layer vs multi-dimensional layer

Please take a look at these Neural Network architectures: I can understand the architecture of Hidden Layer in Net-2 : you add 12 Neurons to your hidden layer... What I can't understand is the 2-...
3
votes
1answer
1k views

Deriving the line for the decision boundary

Let's say I have the following perceptron activation line: $y = 1$ if $\sum_{i}^{d} w_ix_i \geq \theta $ and $y=0$ else. Now, for d = 4, I have derived the following question. $$ w_4x_4+w_3x_3+w_2x_2+...
3
votes
1answer
2k views

What is the correct way of calculating Rectifier Linear and MaxOut functions?

If I have a artificial neuron with 2 inputs: input 1 = 0.7 & weight = 0.7 input 2 = 0.3 & weight = 0.3 If I use a Rectifier Linear (ReLU) as activation ...
3
votes
1answer
492 views

Machine learning: intuition behind perceptron learning algorithm

Given features $x_1...x_n$, weights $w_1...w_n$, calculated output $y = W^T \cdot X$, and actual output $\hat{y}$, the perceptron learning algorithm changes the weights after each iteration as follows:...
2
votes
1answer
2k views

What are the best R packages for a classification problem with use of Neural networks [closed]

Surfing on the internet shows me that there are a lot of different packages and functions which can be used to train neural networks via R. packages such as 'RSNNS', 'nnet','neuralnet', etc. I'm ...
2
votes
2answers
1k views

Visualizing High Dimensional weight space for perceptrons

I am watching the Neural Network videos by Prof. Geoff Hinton. In there he talks about a high dimensional Weight Space for perceptrons. In particular, I am ...
2
votes
2answers
2k views

How to Score an MLP Classifier

I am self-teaching Machine Learning, so excuse me if this is a stupid question. I am trying to understand MLP Classifiers, but would like to know what the best way to create a score is. e.g. ...
2
votes
1answer
238 views

Why cannot my sigmoid classify a linearly separable set with 100% accuracy?

I recently finished my implementation of a feedforward multilayer ANN in Julia. I train it using basic gradient descent with no additions (no regularization, no momentum, no decay, no anything, just ...
2
votes
1answer
1k views

Calculation of decision boundaries with Perceptron

here i will train perceptron and plot decision boundaries (target is generated so I am sure that it is lineary separable). For sake of the example, there is no bias. ...
2
votes
1answer
850 views

Resilient Propagation: How to choose between RPROP+, RPROP-, iRPROP+, and iPROP-

I am using ENCOG to implement a Perceptron network. One of the easiest back-propagation (gradient descent) algorithms to use is the Resilient Propagation algorithm. There are four variants for ...
2
votes
0answers
114 views

Can this network learn the XOR function?

Let's say I have the following constraints: The architecture is fixed (see image) (note that there are no biases) Activation function for the hidden layer is ReLU There's no activation function for ...
2
votes
0answers
74 views

Output node of Perceptron neural network 'learning' unexpected function

I'm building some simple Perceptron networks to gain insight into how they operate. Most of the results are compelling, but there is one that I cannot figure out. Here is my simple Perceptron class ...
2
votes
0answers
355 views

Adding regularization term to perceptron weight update

I hope this isn't a stupid question. I'm trying to reduce overfitting on my perceptron network by adding in a regularization term. However I am not sure where the actual term goes... Usually the ...
2
votes
0answers
1k views

Function approximation using multilayer perceptron (neural network)

I've been asked to solve a problem for a project. I'm working on Python or R. I need to approximate a function with multiplayer perceptron (neural network). The function is: $y= 2\text{cos}(x)+4$ on ...
2
votes
0answers
69 views

Perceptron trained on time series always predicting the same answer [duplicate]

Using the model from theano's tutorial, I'm training a 3-layers perceptron with log returns over a very large dataset (~55,000 points). The output's layer contains two neurons, one for each of the ...
2
votes
0answers
157 views

Machine learning: intuition behind perceptron learning algorithm

Given features x_1...x_n, weights w_1...w_n, calculated output y = Z dot X, and actual output y', the perceptron learning algorithm changes the weights after each iteration as follows: ...
2
votes
0answers
411 views

Deriving step size/learning rate in the hinge loss passive-aggressive/perceptron algorithm

Recall the perceptron algorithm: cycle through all points until convergence $\text{if }\, y^{(t)} \neq \theta^{T}x^{(t)} + \theta_0\,\{\\ \quad \theta^{(k+1)} = \theta^{k} + y^{(t)}x^{(t)}\\ \}$ ...
1
vote
2answers
370 views

Combining perceptrons

Has anyone ever tried to build a Multi-Layer Perceptron Neural Network without the sigmoid function? Let me explain better: We know that a perceptron is a binary classifier that assign the test ...
1
vote
2answers
2k views

Perceptron overfitting?

I'm trying to judge the performance of my perceptron linear discriminant. In one instance I'm training on a sample size of 150 and on another I'm training on a sample size of 1500. I test both of ...
1
vote
1answer
247 views

Derivation of Perceptron weight update formula

I've started out studying Machine Learning and am currently reading up about how a single perceptron works. From the wikipedia page, my understanding is as follows: suppose we have an input sample $\...
1
vote
1answer
80 views

Why do nodes in hidden layer produce different results?

Assuming a simple, fully connected Multilayer Perceptron network with one input layer, one hidden layer with multiple nodes and one output layer. In this case the nodes in hidden layer are ...
1
vote
1answer
839 views

Is the decision boundary of voted/average perceptron linear like basic perceptron?

Is the decision boundary of voted/average perceptron linear like basic perceptron? How can you justify that? It's related to solve binary classification problems in Machine Learning context.
1
vote
1answer
42 views

Perceptron learning for non-linearly separable data

It is well known that perceptron learning will never converge for non-linearly separable data. This means that you cannot fit a hyperplane in any dimensions that would separate the two classes. Is it ...
1
vote
1answer
719 views

Univariate time series multi step ahead prediction using multi-layer-perceptron (MLP)

I have a univariate time series data. I want to do a multi-step prediction. I came across this question which explains time series one step prediction. but I am interested in multi-step ahead ...
1
vote
1answer
94 views

Would multilayer perceptrons be better than multiple regression?

I am using multiple regression to predict the future value of a time series from several other time series. Would doing this with multilayer perceptrons produce better results than multiple regression?...
1
vote
1answer
1k views

Question regarding weight update rule in Perceptron

In the single-layer perceptron, the weight update rule is given by:- $w_j := w_j + (y^{(i)} - \hat{y^{(i)}}) \times x_j^{(i)}$, where $w_j$ is the weight of the $j$th feature, $y^{(i)}$ is the ...
1
vote
1answer
127 views

Combine two single-level perceptrons

Suppose we have a training dataset with $s$ instances, and we split it to 2 sets, of lengths $n, m$ such that $s = n + m$ . We train two perceptron models $p$ and $q$, both are defined by a k-...
1
vote
1answer
285 views

Single Layer Perceptron Question

We have these two mappings: 1) 000→0 001→0 010→0 011→1 100→0 101→1 110→1 111→1 2) ...
1
vote
1answer
542 views

Is Weight Matrix Wij from neuron j to i or from i to j?

I have a question about the Weight matrix from a Neural network. I see different notations for this. I see some publications (e.g. This stanford page) with: Notation 1: $W^{(l)}_{ij}$ denotes the ...
1
vote
1answer
462 views

Can adding an additional feature to a perceptron classifier make the results worse?

I am using perceptron to solve a classification problem. I have a limited amount of features (26) and iterate through all possible combinations of them. A combination of two features [feature_a, ...
1
vote
1answer
904 views

Intuition on upper bound of the number of mistakes of the perceptron algorithm and how to classify different data sets as “easier” or “harder”

I was trying to understand the following result more intuitively (for linearly separable data): $$ k \leq \frac{R^2}{\gamma_g^2}$$ where: k = is the number of mistakes the perceptron algorithm does ...
1
vote
1answer
40 views

Neural Networks Perceptrons and MLPs

While studying as a newbie about Neural Networks I started as everyone from the basics (perceptrons, MLPs) then how backpropagation works before dive in to harder deep learning concepts. Now, I am ...
1
vote
0answers
38 views

Why my perceptron doesn't train well and produces bad results on test data?

I am a newbie in Machine learning and I am writing a small code for Perceptron. This is the first time I am writing code in Python. I have four training data points (X). As they are used for ...
1
vote
0answers
112 views

Computing the Hessian Matrix Diagonal of a multi-layered Feed Forward Neural Network

I am working on using a Feedforward multi-layered perceptron as a function approximator for the pressure distribution of a groundwater system. I am essentially trying to solve a boundary value problem ...
1
vote
1answer
226 views

Over which set of elements should I perform norm clipping of gradients for backpropagation?

I want to normalise the gradients of my multi-layer perceptron in order to avoid the Exploding Gradients Problem, so I thought I would use l2-normalisation but am unsure about how to apply it to the ...