Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [deep-learning]

An area of machine learning concerned with learning hierarchical representations of the data, mainly done with deep neural networks.

2
votes
2answers
8 views

Creating thinned models in during Dropout process

Applying dropout to a neural network amounts to sampling a “thinned” network from it. The thinned network consists of all the units that survived dropout. A neural net with n units can be seen as a ...
0
votes
0answers
19 views

Question about the regularization on Deep learning book (Dr. Goodfellow) [on hold]

In Chapter 8 page 228, I don't understand why the component of $w^\star$ that is aligned with the $I$-th eigenvector of $H$ is rescaled by a factor of $\lambda_i/(\lambda_i + \alpha)$. And in the ...
3
votes
1answer
27 views

What are the benefits of layer-specific learning rates?

I've read about using different learning rates for different layers of neural networks instead of using the same global learning rate for each layer. What's the need for using these different ...
1
vote
1answer
26 views

Are e2e DL systems better than DNN-HMM models in speech recognition?

End-to-end deep learning systems for automatic speech recognition (ASR) have been around for a while now since Deep Speech (2014), but I noticed that DNN-HMM based methods are still performing well ...
1
vote
0answers
12 views

How do backbone and head architecture work in Mask R-CNN?

In this diagram, we see the two convs. It is said that these convs are a part of the Fully Convolution Network (FCN). In their paper Mask R-CNN (He et al., 2018), they mentioned something about the ...
1
vote
1answer
28 views

Neural Networks Perceptrons and MLPs

While studying as a newbie about Neural Networks I started as everyone from the basics (perceptrons, MLPs) then how backpropagation works before dive in to harder deep learning concepts. Now, I am ...
0
votes
0answers
36 views

Emperical evidence if a model has too much parameters it will never overfit [on hold]

This Question stems from the comments on this answer, and also this answer, the premise that is presented by User Vlad is if you have way too much parameters, your model will not overfit (What comes ...
1
vote
1answer
26 views

Neural Networks - Difference between 1 and 2 layers?

I'm currently working on a regression problem, using neural networks to constrain parameters for a complex physical scenario. I am searching the hyperparameter space for the best model and have thus ...
1
vote
0answers
8 views

Which is the more reliable method for reporting classification results in deep learning?

I have two methods to compare, the one which reports weighted F1-score with imbalanced data and, the one which reports better F1-score with balanced data. I am confused as to which method's ...
0
votes
0answers
9 views

Binary target prediction using LSTM with sparse events in time

I have a data of patients that have multiple events happening in there medical history, I'd like to predict a target of having a specific targeted-event in the next 30 days. The data is timestamped ...
0
votes
2answers
16 views

Image classification using Semantic Segmented Images

Can we use the semantic segmented images directly to perform image classification using CNN model? Updating the question: I am trying to classify images as below: a. Input : Images taken from camera ...
0
votes
0answers
13 views

How can I fine tune simple RNN or LSTM? [on hold]

I'm dealing with RNN and LSTM models by normalized data in range of [-1,+1] and reshaped data for each time sequence from 3 individual matrices A,B,C to long row includes elements of all 3 matrices ...
0
votes
1answer
14 views

Sample complexity of deep reinforcement learning agents on smaller state spaces versus zero-padded state spaces

If I train two agents, one on environment A and one on environment A', where A' is just environment A padded with 10 rows of zeros, what can I predict will happen in terms of relative sample ...
0
votes
2answers
39 views

Neural Network for input values optimization

I have electric machine, which parameters I measure by 10 sensors. 8 of them measures "input" values and 2 of them result (output). I've got tons of historical data of all of these sensors. I built a ...
1
vote
0answers
17 views

Apply K-means to the columns of the covariance matrix

In Section 5.3 of the paper distilling the knowledge in a neural network, it says we apply a clustering algorithm to the covariance matrix of the predictions of our generalist model, so that a set ...
0
votes
0answers
6 views

Clarification on notation used to present back propagation algorithm in 'The Deep Learning Book'

In the deep learning book (free version is available online) the backpropation algorithm is explained in section 6.5. I have a question on equation (6.53): $$\frac{\partial u^{(n)}}{\partial u^{(j)}}...
0
votes
1answer
13 views

Encoding Layers in the Transformer

In the transformer architecture for NLP, at each layer there are multiple self-attention filters. My question is about the encoded content within these filters. An example can be found here: My ...
3
votes
1answer
47 views

what exactly happens during each epoch in neural network training

1.Across different epochs, which of the following is/are updated? initial weights (initial ConvNet filter matrices, initial fully connected weights) hyper parameters: number of ConvNet filters, size ...
0
votes
0answers
13 views

Do γ and β “undo” the effects of batch normalization?

Let H be a minibatch of activations for a layer to be normalized, where activations of each example are in a row of the matrix, and each column represents the activation of a given unit in the layer. ...
0
votes
0answers
16 views

how to prepare text data for LSTM autoencoder

My main goal is to come up with some topics using LSTM autoencoder. I want to use 20 news_group data set. after reading lots of material and looking at some GitHub project, I am still not clear how ...
0
votes
1answer
13 views

Understanding semi supervised technique called mean teachers

I am trying to understand applying semi supervised learning as described in this paper. Describing the final recipe as described in this paper: Take a supervised architecture and make a copy of it. ...
0
votes
0answers
20 views

IBM new Floating Point (FP8) DNN training format in NeurIPS2018 [closed]

IBM uses FP8 in the forward pass and FP16 in the backprop. with stochastic rounding scheme. What do you think of this paper? Will it scale to train networks larger than ResNet50 (what they report in ...
0
votes
0answers
21 views

On the Relationship between Data Size, Number of Epochs, Number of Iterations and Convergence of a Model

I did the following two experiments with a model on a dataset: Experiment 1: Training on a small dataset (~50 examples) The model took around 60 epochs to overfit just this small dataset. Each epoch ...
0
votes
0answers
10 views

Shapes of input and outputs for LSTM architecture?

I have a sequence data like X1, X2, X3, X4, X5 -> y1,y2,y3,y4,y5 X6,X7,X8 -> y6, y7, y8 Where Xi is m x n dimension matrix, n is the number of columns (...
0
votes
0answers
8 views

evaluation result is bad on training dataset [closed]

I train a dnn regressor model and use the trained model to evaluate the training dataset. The training loss is low however the evaluating loss is very high. I am confused by this. As I know, even if ...
0
votes
0answers
12 views
+50

on the understanding of visualization of the fully connected layer

When the final fully connected layer is visualized, it shows a big picture consisting of mini-pictures patching together and overlapping with each other. The mini-pictures are visibly recognizable to ...
0
votes
1answer
40 views

How is the loss(Backpropagation) for simple RNN calculated when dealing with batch?

I have been trying to implement a simple RNN in Python. I saw Andrew Ng's course on RNNs, and then I tried to write one for myself. However, it seems I have not ...
0
votes
0answers
4 views

Possibility of constrain the product of norm of weights across all layers

Problem One theoretical result I read says that the generalization error of deep neural networks could be independent of network depth and width when the product of norm of all weights across all ...
0
votes
0answers
30 views

Backpropagation Algorithm not

Every single derivation of the BPA I have ever seen uses particular loss functions as example. The derivative is computed with respect to the weights and then a recursive relation is found. Now while ...
0
votes
0answers
6 views

Training Neural Network at Decile Level

I have a simple feed forward neural network regression model that I'm training on customer data to predict their usage amount. The MAPE is above 50%. The data is heavily skewed and when I log ...
0
votes
0answers
5 views

How to train a CNN model with variable input image sizes?

I am currently trying to make a model that is general enough to handle variable input sizes. Specifically, I am training a model to segment MRI images, but MRI images are variable in its resolution ...
0
votes
1answer
19 views

Correct algorithm for string classification

I have a long list of DNA strings (of equal length) made of 4 letters (A,T,G,C). I want to do a binary classification on the strings. I have two basic quetsions: I have a lot of duplicate strings ...
2
votes
1answer
31 views

Creating a neural network that can make a decision with optional arguments

I'm a final year computer science student and for my final year project I have to design a neural network to play a little known board game called 'The Downfall of Pompeii'. I have to use ...
1
vote
1answer
32 views

Changing the training/test split between epochs in neural net models, when doing hyperparameter optimization

Consider a predictive modeling case where the number of samples is limited, but the data on the samples is rich. For context, I'm doing a multivariate time series prediction, with a few thousand (...
2
votes
1answer
41 views

Catastrophic forgetting: Retraining a trained neural network with small data

I have a fully connected deep neural network with 7 hidden layers, which is trained with around 20000 simulated materials data. And we've got a very small measurement dataset (size<200) which ...
0
votes
1answer
22 views

Reinforcement learning based Q-learning for wireless routing

In the Q-learning method to get the optimal strategy, the update method is like the following: \begin{equation} Q(S,A) \leftarrow \ Q(S,A) + \alpha [R+\gamma~max_a(Q(s',a)) -Q(S,A)] \end{equation} If ...
0
votes
0answers
9 views

Distinguish between real video and deep learning generated video [closed]

There is an authentic live streaming video coming out from a source (streamed to the internet). Say, someone else, like a spy agency, wants to sabotage this streaming. They intercept and replace the ...
2
votes
0answers
23 views

What is the number of filter when using CNN for sentence classification

I am new to machine learning and NLP. During reading convolutional neural networks for sentence classification I'm having trouble understanding it. In the paper it says that a feature map c has ...
2
votes
1answer
20 views

What is mode-averaging in wake-sleep algorithm?

I was reading Hinton's paper on Deep Belief Nets, A Fast Learning Algo for DBNs. In the introduction section, the authors write: Section 5 shows how the weights produced by the fast greedy ...
2
votes
0answers
102 views

Can this network learn the XOR function?

Let's say I have the following constraints: The architecture is fixed (see image) (note that there are no biases) Activation function for the hidden layer is ReLU There's no activation function for ...
3
votes
1answer
103 views

Cannot understand LSTM inference

I seem to have stumbled on a hole in my understanding around LSTMs. In short, I cannot understand how even a simple one is actually fed samples, upon inference time/training time. Here are the details:...
0
votes
0answers
32 views

Using temperature as feature in Neural Network

I'm currently putting together a Neural Network for doing Sales Forecasting for hundreds of products. The domain experts know that the sales spike when the temperature drops and so I started to use ...
0
votes
0answers
7 views

event detection with several sensors

I have several of sensors. each sensor shares some detection area with his neighbors. each sensor output goes to simple deep learning network to provided a score (event happened or not on a single ...
2
votes
1answer
50 views

What happens to the initial hidden state in an RNN layer?

I thought I knew how RNNs work, however, when I tried to actually implement it myself, I faced some issues. For one, how do we deal with the initial hidden state? ...
2
votes
0answers
42 views

Validation loss increases while Training loss decrease

I am training a model and the accuracy increases in both the training and validation sets. I am using a pre-trained model as my dataset is very small. I am not sure why the loss increases in the ...
0
votes
0answers
20 views

Attention for short sequence length. Is it reasonable?

Will the attention mechanism be useful for the short sequence length? Let's say your training corpus has each query of MAX length 10. and most queries are of word length 3-4 words. How reasonable is ...
1
vote
1answer
26 views

CNN training in bfloat16

Are there any efforts so far for training CNNs end-to-end with bfloat16 format? especially the convolution part, i.e. both multiplication and addition is done in bfloat16. Can this scale to large ...
1
vote
1answer
15 views

Differentiable method to convert voxel representation into pointclouds?

I'm finding a way to convert 3D voxel data into 3D point-clouds. Since I'd like to do it inside a deep-learning architecture, the conversion has to be differentiable. Is there such a method? Thanks ...
0
votes
1answer
10 views

Neural architecture search dataset

I have recently started looking more into autoML, where we have a "Controller" system, who outputs architectures and hyperparameters, and is given a reward based on the performance of a system trained ...