Artificial neural networks (ANN), are composed of 'neurons' - programming constructs that mimic the properties of biological neurons. A set of weighted connections between the neurons allows information to propagate through the network to solve artificial intelligence problems without the network ...

learn more… | top users | synonyms

0
votes
0answers
8 views

How to find derivative of softmax function for the purpose of gradient descent?

I'm trying to understand back propagation algorithm for multiclass classification using gradient descent. The output layer is a softmax layer, in which each unit in that layer has activation function: ...
0
votes
1answer
26 views

Understanding Matrix and Vector Notation

I am trying to understand the Matrix and Vector Notations on page 2 here: (the page is also pasted below, to make it easier to explain the problem). Problem: For equation (2), I think it should be ...
0
votes
1answer
16 views

Cross entropy-equivalent loss suitable for real-valued labels

I am building a model whose outputs are between 0-1 and the goal is to minimize a cost function over the predicted values and labels. So far everything seems to be easy but my labels are real-valued ...
1
vote
0answers
31 views

What is an explanation of the example of why batch normalization has to be done with some care?

I was reading the batch normalization paper[1] and it had one section where is goes through an example, trying to show why normalization has to be done carefully. I honestly, can't understand how the ...
1
vote
1answer
25 views

CNN. Why convolution and composition? [on hold]

About convolution: prof. Brad Osgood said during the course EE-261 said that we can not fully "visualize" convolution. E.g. https://see.stanford.edu/materials/lsoftaee261/book-fall-07.pdf , p.105: ...
3
votes
1answer
58 views

Why does Bengio's deep learning theory book claim $\hat{y} = x w_1 … w_i … w_l$ is a non-linear function of $w_i$?

In chapter 8 section 8.7.1 it tries to explain batch normalization. In the second paragraph of that section it tells us to consider the simple example: $$ \hat{y} = x w_1 ... w_i ... w_l $$ and ...
0
votes
0answers
5 views

Spatial structure of CNN features

Is there any work to study the space of features learned by convolutional neural networks like if they perhaps lie on a manifold? What about other representations like SIFT?
0
votes
0answers
14 views

Intuition behind error updating for inner layer neurons

In backpropagation the usual way to calculate the amount of error of a layer is $$\delta_0 = y_{expected} - y\\ \delta_i = \sigma'(input)\sum_{j \in outputs(i)}{\delta_jw_{i,j}}$$ where $\sigma$ is ...
0
votes
1answer
10 views

Default Batch Size in R NNet package

I'm using the nnet package in R. One of the parameters is "maxit" but there is no batch size parameter. As such, I am confused. Is an iteration one pass through an entire data set? Or is the batch ...
0
votes
2answers
32 views

Where in R code should I us set.seed() function (specifically, before shuffling or after) ?

I've been using the set.seed() function to reproduce same results on multiple runs. However, I don't understand where to use the function. the reason I'm asking this is because if I use the function ...
1
vote
0answers
12 views

Can we obtain probability distribution of a repeatable event using Neural Networks?

We have a data where input of every sample corresponds to how many times a dice is rolled. The output is the sum of all the outcomes. For instance the data is ...
0
votes
0answers
20 views

Neural Networks to Explain User Behavior

In the past I've used logistic regression to not only predict user behavior, but to understand the relative importance of each variable to help the business teams understand which levers they can pull ...
1
vote
0answers
4 views

Computing range of threshold values for given inputs and classifications

I'm having some trouble with the following type of exercises: Given a feed forward neural network with Heaviside activation function (the function that outputs 1 if the input is >= some threshold ...
0
votes
0answers
31 views

Neural Network with General Topology

I would like to construct a neural network that has $n+m$ inputs and $k$ outputs. For example $n=5$, $m=2$, $k=2$. The topology is as follows: there are several layers that process the first $n$ ...
0
votes
0answers
17 views

Why classification accuracy in validation set gets lower if validation cost also gets lower?

I'm training neural network for some simple classification task using tensorflow and have 2 output neurons, using softmax classification. My question is why accuracy on validation set gets lower when ...
2
votes
2answers
51 views

Is it possible to train neural network to draw picture in a certain style?

Is it possible to train neural network to draw picture in certain style? (So it takes an image and redraws it in a style it was trained for.) Is there any approved technology for such kind of a ...
0
votes
0answers
11 views

How to mix known and unknown features in timeseries RNN?

Consider the following data: ...
0
votes
0answers
18 views

To what extent has functional analysis been used in understanding neural networks?

Have the family of models encapsulated by neural networks been studied from a functional analysis perspective? Are there any general theorems, results, etc. about neural networks from this ...
0
votes
0answers
40 views

log bilinear language models architecture

I am trying to understand this paper. The paper basically introduces a simple variation of feed forward neural network with $h$ hidden units, in which inputs are a sequence of words $(w_1, ...
1
vote
1answer
31 views

Can function parameters be the output of a Deep Neural Network

I'm just starting out with Tensorflow and DNNs. My question is, can some parameters that make a function (e.g. control points that can make up a Gaussian function) be the output of a DNN? What I'm ...
0
votes
0answers
52 views

who can explain the gradient descent to a not -mathematician [closed]

I'm trying to get a better understanding of a neural network backpropagation neural network and especially the gradient descent. I'm not a mathematician so I can't read or understand the formula's who ...
0
votes
0answers
44 views

Intuition behind using Noise Contrastive Divergence in neural language models

I am going over this paper which uses NCE to avoid dealing with the normalization constant of a log bilinear model, when maximizing the likelihood. The problem is clear, but I don't understand what is ...
0
votes
0answers
14 views

Neural Network - Learning accuracy drops heavily after a couple of epochs

I designed a neural network to classify some images into 28 classes. Here are the parameters : Weight Decay : 0.005 Momentum : 0.01 Learning Rate : 0.001 and 0.005 Learning Decay : 1 Input : 100x100 ...
0
votes
0answers
7 views

How many parameters required to approximate quadratic function with NN

How many layers do you need to build a RELU network (1-layer or deep) to approximate x^2 function on [0,1] with 1e-6 accuracy. What is the practical result on the same, say with Tensorflow ...
0
votes
0answers
8 views

A conceptual question about LSTM-RNN

I am working on series prediction by LSTM-RNN. In the training stage, I use a random series (white noise ) as input to go through a system and get the output. LSTM is implemented to learn the ...
-1
votes
0answers
24 views

Converting from strings to factors in R [closed]

I am having a problem converting from strings to factors. This is my code and in it is how I tried to convert from strings to factors. ...
1
vote
1answer
22 views

When to run truncated backpropagation through time in recurrent neural networks?

I'm interested in training recurrent neural networks using truncated packpropagation through time (BPTT). From Sutskever (2013): Truncated BPTT...processes the sequence one timestep at a time, ...
0
votes
0answers
15 views

More than one output neuron firing [closed]

I am building a neural network to solve a multi-class classification problem. Is it possible for more than one output neuron to fire?
0
votes
0answers
7 views

RBM with float units in visible layer

Currently I have been studying RBMs and concluded that it could be good model for my purpose. In my case Visible layer should be real valued of both signs, basically it is just word2vec values (which ...
0
votes
0answers
16 views

Preparing the data for neural network

I have 2 categorical variables Make and Model as inputs. I am trying to imagine how the data will look like. I am going to pass a training example through the network that is a vector that gives a 1 ...
3
votes
1answer
62 views

What are the effects of depth and width in deep neural networks?

How does depth and width in neural networks affect the performance of the network? For example, He et al. introduced very deep residual networks and claimed “We obtain [compelling accuracy] via a ...
1
vote
1answer
20 views

choosing between traditional statistical models and neural networks using RMSE

I am new to neural networks, and am more familiar with classical linear regression type models. I have a basic question about choosing between the two in attempting to develop a predictive model. Is ...
0
votes
0answers
24 views

Are Residual Networks related to Gradient Boosting?

Recently, we saw the emergence of the Residual Neural Net, wherein, each layer consists of a computational module $c_i$ and a shortcut connection that preserves the input to the layer such as the ...
3
votes
1answer
54 views

Neural Networks Vs Structural Equation Modeling What's the Difference?

I'm studying about artificial neural networks (ANN) for the first time and I am struck by how the concepts of neural networks appear to be similar to structural equation modeling (SEM). For example, ...
0
votes
1answer
34 views

Neural Networks back propagation

I have gone through neural networks and have understood the derivation for back propagation almost perfectly (finally!). However, I had a small doubt. We are updating all the weights simultaneously, ...
0
votes
1answer
238 views

Does the skipgram language model try to predict all context words at the same time?

In the skipgram language model (Mikolov et al., 2013), a neural network with one hidden layer tries to predict surrounding words from current words of the corpus. After training, the hidden activation ...
0
votes
0answers
12 views

“Temperature” for generating continuous functions in RNNs

I have been playing around with RNNs for the last week. I have done mostly text generation. In my text generation programs I have a sample function which I found online. ...
1
vote
0answers
28 views

Is it normal that a Neural Network sometimes doesn't learn Xor?

I've implemented a neural network and I'm training it to compute Xor. 1 out of x times it fails to learn, where x is about 5 or 10. It then gives e.g. 0.67 instead of 0 as output for input (1,1). Is ...
0
votes
0answers
16 views

In a NN, how to calculate variance of gradient using backpropagation

Supposing I know the values of the loss function (f), and its gradient over all parameters (df), I can calculate the variance (estimated noise) of f, but how do I calculate the variance of df (over ...
0
votes
0answers
8 views

Architecture of Input layer for categorical variables

Suppose I have three categorical variables make, model, and year. I am trying to find how many neurons I should have in my input layer. Suppose Make contains three elements: Honda, BMW, and Mazda. ...
1
vote
0answers
24 views

Steps to designing a neural network [closed]

What are the logical steps in designing a neural network? This will be my first and I am still learning the 'language' of neural network (i.e reading Michael Nielsen's online book). How should I ...
1
vote
1answer
55 views

Number of neurons in the output layer

In a classification problem, how do you decide on the number of output neurons you have in your neural network? Is the number of neurons equal to the number of classes you have? Is there a limit on ...
3
votes
1answer
56 views

Indoor location using WiFi Signals and Machine Learning

I am trying to determine in which zone of a building a person is located based solely on the strength of the WiFi signals her cellphone gets. Currently, we are making measures with an Android App, for ...
0
votes
3answers
36 views

Neural network - meaning of weights

I am using feed-forward NN. I understand the concept, but my question is about weights. How can you interpret them, i.e. what do they represent or how can they be undestrood (besied just function ...
0
votes
1answer
57 views

Why is optimisation solved with gradient descent rather than with an analytical solution? [duplicate]

I'm trying to understand why, when trying to minimise an objective function, gradient descent is often used, rather than setting the gradient of the error to zero, and solving it analytically. In ...
0
votes
2answers
49 views

Neural Network with unknown number of Neurons in output layer

Is it possible to design a network with an unknown number of neurons in the output layer? I am trying to solve a classification problem, where I use motorcycles' exterior color, interior color, and ...
0
votes
0answers
8 views

incorporating the concept of 'coverage' in cross entropy loss

by coverage, I mean that I want my binary classifier (neural network) to perform EXTREMELY well on a large portion of the data (e.g. 95%), even if this means that it performs extremely poorly on the ...
0
votes
0answers
22 views

How to predict on part of image after training on other part of image?

I have images of identity cards (manually taken so not of same size) and I need to extract the text in it. I used tesseract to predict bounding boxes for each letter and am successful to some extent ...
0
votes
0answers
15 views

optimizing 2 factors - Can you use NSGA-II to optimize this?

I have 3 machines A, B and C. I would like to rank the machines based on which machine maximizes Score1 and Score2. Score1 and Score2 are performance measures that rang from 0-100%. Below is some ...
0
votes
1answer
20 views

Autoencoder with tied weights: bias?

For some unsupervised learning problem, I need to train an autoencoder, so that I only have to store the encoder afterwards. However, I am not sure on how and if the bias weights can be tied. To make ...