Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Artificial neural networks (ANN), are composed of 'neurons' - programming constructs that mimic the properties of biological neurons. A set of weighted connections between the neurons allows information to propagate through the network to solve artificial intelligence problems without the network ...

1
vote
1answer
14 views

Validate implementation of back-propagation algorithm

Let's say I implemented a CNN. Is there an easy way I can validate, that my implementation of back-propagation does not contain errors ? May be I can feed some dummy values into my network so it can ...
0
votes
1answer
12 views

Will binary model be automatically mirrored if train set inverted?

Suppose we are training binary classifier to output 0s and 1s. It is trained and now returning float values y in range from 0 to 1. Now suppose we inverted the ...
1
vote
0answers
5 views

How to handle variable size input data (incomplete) to build/train a NN for regression?

Suppose you have the classical example of predicting house prices and you have as input features area size, built year, number of previous owners, city, number of floors, number of bedrooms, etc. But ...
0
votes
1answer
31 views

Question regarding machine learning models in production

for example, i have a feature with 5 distinct values and once one hot encoded this becomes 5 columns, but lets say the data that needs to be predicted has 4 distinct values, the neural network won't ...
0
votes
0answers
15 views

Need help with Simple Neural Network gradient

I have the following expression below: $f_{1}=aW_{1}+b_{1}$ $m=g(f_{1})$ where g is a sigmoid function How can I calculate the gradient of $\frac{dm}{dW_{1}}$ and $\frac{dm}{da}$ ? I know that $\...
0
votes
0answers
16 views

how to choose model when training accuracy is lower than validation accuracy while training neural network?

Below is a specific case but a general situation i find myself involved with in my job. This question is intended at getting ideas on how to pick the best model: Dataset: rows: 10,166, features: ...
0
votes
0answers
14 views

t-sne V CNN extracted features

If one were to visualise images (say MNIST) with t-sne we get great separation in the t-sne lower dimensional space. However what would happen if one where to run a CNN on MNIST, then remove the ...
2
votes
1answer
24 views

Are RNNs inherently flawed? Supervised Learning assumes IID data but sequential data is not IID

From what I understand, Supervised Learning operates under the assumption that the data is I.I.D. It seems to me that the training procedure for RNNs is flawed. We receive observations in a sequential ...
0
votes
0answers
13 views

Different results for stratified cross validation and train test split for CNN

I am trying to develop an image classifier using conventional neural networks. Now I was looking at evaluating the model using 10-fold cross validation and through repeated train test splitting (hold ...
0
votes
0answers
20 views

AlexNet Paper dimensions does not match?

I am following the paper where AlexNet was introduced, and the dimensions they report just don't match with the figure they attached. The output of the first conv layer (which is 96 11x11x3 ...
1
vote
1answer
26 views

Neural networks to predict a nonlinear curve

I want to model a complex nonlinear function using neural networks (keras). Training data: input - 8500 x 176 matrix of features, output - 8500 x 8 matrix, each row corresponds to 8 points which ...
0
votes
1answer
21 views

How can we interpret the learning curve including loss for training and test in a deep learning model?

I am working on 3D medical image segmentation area. It may take 2-3 days to finish one round of training. How can I interpret the learning curve if over-fitting is happening or not? It happens to me ...
1
vote
2answers
34 views

In Recurrent NN, what's the reason for adding instead of multiplying the input term and the state term in the hidden units?

As we know, the hidden layer unit has the following activation: $$h_t=tanh(UX_t+Wh_{t-1})$$ So there is the interaction between the input and the previous state: $UX_t+Wh_{t-1}$. My question is why it ...
1
vote
2answers
27 views

Mathematical structure of SimpleRNN in keras

Two types of RNN can be used: Type1: The output is being used as state h(t) = g(W1.x(t) + W2.h(t-1) + b1) Type2: There is a state in addition to the output a(t) = g(W1.x(t) + W2.a(t-1) + b1) h(t) ...
0
votes
0answers
11 views

Dummy Variables in Neural Networks and Random Forests

In logistic regression, dummy variables are included in the model with k-1 categories. However, I am not sure how to deal with dummy variables using NN, RandomForests, DecisionTrees, and other ...
0
votes
0answers
23 views

PHD topic in Macro economic analysis using Machine learning or network science [on hold]

I am looking for a recommendation to do Ph.D. proposal in dealing with Macro_Economic analysis that uses either Machine learning/ neural science or network science tools. it seems a bit confusing for ...
1
vote
1answer
63 views

Does Neuton framework offer better ML algorithms than the ones currently known to the public? [on hold]

I came across an article about new proprietary 'Neuton' framework. I can't find educated opinions from professionals in the field. What the framework promises feels 'too good to be true'. Is it?
0
votes
0answers
10 views

Mean Test Set Performance of LSTM & Evaluation

In the paper "Greff et al, 2017 - LSTM A Search Space Odyssey" they evaluate different variants of LSTM architectures against different tasks/datasets. Could you help me to understand the evaluation? ...
0
votes
0answers
9 views

Can a neural network have an activation function that is a transformation of the parent function?

Neural networks can have activation functions like a tanh(x), a sigmoid function, ReLU, etc.. But can we have an activation function that is a transformation of any of these functions? For instance, ...
3
votes
1answer
49 views

Intriguing properties of neural networks

In a paper by Ian GoodFellow1, on page 3, what is meant by: Our experiments show that any random direction $v ∈ \mathbb R^n$ gives rise to similarly interpretable semantic properties. More formally,...
0
votes
1answer
15 views

Testing an LSTM making predictions 1 timestep into the future

Say I have a time series data set of 100 sequential timesteps, and I want to train and test an LSTM on the data set, but only on forecasting a single timestep into the future. I want more than one ...
0
votes
0answers
7 views

Support Vector Machines VS LSTMs: How well it is justifiable to use LSTM for its Generalization properties?

The question is pretty straightforward, How well one can justify using LSTMs(Neural Networks) for text classification task in terms of "Generalization" compared to classic support vector machines(SVM) ...
0
votes
0answers
5 views

Normalizing Input data in neural network using matlab

I have a data set with 8 inputs and one output. I want to train this data set with Neural Network modeling in MATLAB. When i input data without normalization, the MSE is very large around 1e+3. But, ...
0
votes
0answers
12 views

Softmax Gradient Help

I am trying to calculate the gradient of a neural net. Here is the net. Its a 1 layer net with softmax $f_{1}=xW_{1}+b_{2}$ $Y^{*}=S(f_{1})$ $E=-\Sigma_{i}\left(y_{i}log(Y_{i}^{*})\right)$ I am ...
0
votes
0answers
5 views

What hyperparameters should be sampled (together) for neural networks?

I'm using a neural network for a multi-target regression task and would like to perform hyper-parameter optimization. The network has one hidden layer and uses MSE loss on the output. I have large ...
10
votes
1answer
272 views

Does the function $e^x/(1+e^x)$ have a standard name?

Does a function in the form $e^x/(1+e^x)$ have a standard name? E.g. $y = a + bx$ is a linear function.
0
votes
0answers
7 views

Getting VGG features from FC1 layer [closed]

I have tried to implement and train the VGG16 from scratch on 2 images coming from cifar10 dataset for experimental purpose I have tried to get the features of the input image from the first fully ...
5
votes
1answer
298 views

Is there any paper which summarizes the mathematical foundation of deep learning? [closed]

Is there any paper which summarizes the mathematical foundation of deep learning? Now, I am studying about the mathematical background of deep learning. However, unfortunately I cannot know to what ...
0
votes
0answers
17 views

Have CNNs matched radiologists in predicting breast cancer from mammograms yet?

I'm at a talk right now about machine learning in medicine. One of the slides showed a CNN (Convolutional Neural Network) rivaling radiologists in accuracy of mammogram readings. It predicted things ...
0
votes
0answers
20 views

Single words as input in a feed forward neural network?

Is there a way I can input single words, e.g. names, as input in a feed forward neural network? It has to be a Feed forward NN, so I guess I have to implement some sort of pre-processing, like ...
0
votes
1answer
20 views

What do the results of tuning β1, β2 in ADAM imply about the gradients or the data?

For example: If I have a simple 3-layer neural network that demonstrates better performance on the test set when the value of β2 is .95 when compared to the default .999 over several trials of cross-...
0
votes
0answers
14 views

Training an autoencoder with reconstruction target and classification labels

I would like to binarily classify a number of sequences which contain heavy noise. For each of these sequences, I have another sequence which is related to it, contains less noise and is also closely ...
0
votes
0answers
23 views

Neural Networks with ReLU activation function

I was trying to write an explicit parametrisation of all functions $f$ : $\mathbb{R}$ $\to$ $\mathbb{R}$ that can be represented with a feed forward neural network with two hidden layers with 10 ...
0
votes
0answers
15 views

Why does the minimax cost of GAN`s generator looks like this?

Here is a graph of the generator's cost of GAN, which is taken from Goodfellow`s paper (https://arxiv.org/pdf/1701.00160.pdf). The formula for the minimax cost of the generator is: Why does the ...
0
votes
0answers
9 views

LPP vs Convex optimisation for Neural Networks

I am studying for my statistics exam and we have Linear Programming,Simplex and graphical method, Duality Programming and Integer Programming. I have a great interest in Convex optimisation and wanted ...
1
vote
0answers
22 views

What does the ADAM algorithm actually guarantee?

Can someone provide an intuitive explanation as to what the propositions of the ADAM algorithm actually guarantees? and whether they are strong guarantees? https://arxiv.org/pdf/1412.6980.pdf There ...
0
votes
1answer
24 views

Common Neural Network practice [duplicate]

I am in the early stage of studying the Neural Network. Here are the list I made during the online classes. Shuffle the data Normalized the data by ...
1
vote
2answers
36 views

Why is it possible to train a semantic segmentation neural network like U-net/Tiramisu from scratch using small data-set like few hundreds

Why is it possible to train a semantic segmentation neural network like U-net/Tiramisu from scratch using small dataset like few hundreds. While for the image classification task, it is not ...
0
votes
2answers
23 views

How to calculate the derivative of crossentropy error function?

I'm reading this tutorial (presented below) on computing derivative of crossentropy. The author used the loss function of logistic regression I think. https://www.dropbox.com/s/rxrtz3auu845fuy/...
1
vote
0answers
11 views

how to do convergence analysis for Generative adversarial network? [closed]

i have two variants of Generative adversarial networks. How to compare their performance with respect to converge?
0
votes
0answers
4 views

Hopfield network dynamics with 1/0 vs +1/-1

Hopfield networks can use either 1/0 for the node activation values, or +1/-1. Nobody ever seems to discuss what difference this makes. They just pick one and run with it. Clearly, the dynamics are ...
1
vote
1answer
27 views

Can Neural Network take multidimensional inputs?

I recently learned RNN and find that a common feature of it and CNN is that they use either a LSTM cell or Convolution to process a single multidimensional input (like images and word embeddings which ...
0
votes
0answers
15 views

How often a Recurrent Neural Network should retrain?

I am working on a recurrent neural network (LSTM) to predict a time series. since the training takes time, I have some concerns about the real-world situation when the model is going to implement in a ...
0
votes
0answers
11 views

GAN and NN for sparse data

I have a set of images which represent some correlated sparse data $x_1,\ldots ,x_n$. there are a number of specific pixels in the images which might hold value or not (with some probability), while ...
1
vote
0answers
11 views

Interpreting accuracy graph for a LSTM model | Keyword Prediction

I have created a LSTM model for keyword prediction. I am using RMSE optimizer for training. I observe that the train and test accuracy decreases at first and then fluctuates without much difference in ...
0
votes
1answer
17 views

“Forward Propagation” in Neural Networks

In backwards propagation, one tries to minimize the cost function in the most efficient way by looking at small changes in the bias, the previous activation, and the weights of a neuron. We start in ...
0
votes
0answers
27 views

Network's loss is steadily decreasing, accuracy remains at chance level

I'm training a Resnet v2 straight out of the TF repository on cifar-10 and I'm getting weird results. The training loss starts out at ~2.4 and steadily decreases into the 1e-4 domain. However, ...
1
vote
1answer
44 views

“Denoising” autoencoder with distortions other than Gaussian noise

I watched some talks by Yoshua Bengio, where he often refers to denoising auto-encoders (AE) as a powerful method to learn (unsupervised) representations on an input space (e.g. here). The idea as ...
0
votes
0answers
10 views

Clarification of examples used in WGAN paper

I wonder if anyone could help me understand the following slide What does it mean for $P_0$ to be a distribution of $(0, Z)$? I am given that $Z$ is sampled from a uniform distribution, hence I ...
0
votes
2answers
44 views

Can a neural network learned function $f$ be used for calculating its derivate $f'$?

$y$ is a scalar value, $x$ is an $m \times 1$ vector. Function $f$ is a map $y = f(x)$. Say we have $n$ examples {$(y_1, x_1), (y_2, x_2), \cdots, (y_n, x_n)$}, and we use these examples for fitting ...