Questions tagged [residual-networks]
A type of neural network architecture suggested in 2015 that allows to train very deep networks with 100s or 1000s of layers.
48
questions
2
votes
1
answer
20
views
No activation function between two convolutional layers in MUNIT?
I'm reading the code of NVIDIA's MUNIT, the code of the resnet is as follows:
...
- 21
0
votes
0
answers
9
views
How the skip connections occur in resnet50 architecture when input output layers are different? [duplicate]
I am having difficulty understanding how the skip connections occur in resnet50 when the input and output layers are different in shape.
For example, in the first residual block, a 56*56*64 size ...
1
vote
0
answers
25
views
Strange high values in the probability vector in Image Classification
I have successfully trained a ResNet50V2 model. I used transfer learning on ImageNet database, and launch the inference on the unseen data. I obtain a high accuracy (>80%), but when I print the ...
- 51
1
vote
1
answer
26
views
Does it make sense to use Residual Blocks in a simple FFN classifier without convolutions?
I'm looking for ways of fine tuning my FFN binary classifier, which operates on large, flat vector inputs. The theory behind ResNets seems promising to me, but I'm not confident about whether they are ...
- 75
1
vote
0
answers
36
views
How does the fixed point interation in invertible resnets work?
I feel like I am missing some easy point about this invertible resnet paper which is making it hard for me to grasp how the fixed point iteration works.
stated simply, the residual connection in a ...
- 730
0
votes
0
answers
1k
views
How to architect an encoder-decoder architecture using ResNets?
I am trying to implement my own encoder/decoder architecture in Pytorch. Specifically I am trying to use ResNet-18, both for encoding and decoding part. While by following the paper I believe I ...
- 213
1
vote
0
answers
45
views
Backprop in Residual neural network?
I'm trying to build a Residual neural network with 2 layers , and I'm having difficultiy understanding what are the equations for the backprop for the following :
$$
W_{2}x+tanh(W_{1}x+b_1)+b_2
$$
...
- 11
1
vote
0
answers
185
views
Derive the gradient matrices w.r.t. W1 and W2 and backprop equation in a Residual Network [closed]
How would I go about deriving gradient matrices w.r.t. W1 and W2 and backpropagation equation in a residual block that is a part of a larger ResNet network with forward propagation expressed as:
$$ F(...
- 11
15
votes
1
answer
8k
views
Why are residual connections needed in transformer architectures?
Residual connections are often motivated by the fact that very deep neural networks tend to "forget" some features of their input data-set samples during training.
This problem is ...
- 287
4
votes
1
answer
710
views
Is it necessary to skip two or more layers in ResNets?
I was watching Andrew Ng's video about ResNets and in that video he skipped 2 layers while "short cutting" them. For example, if we have input $a^{l}$, next layer will be $a^{l+1} = g(z^{l+1}...
- 163
6
votes
1
answer
1k
views
Are there any weight matrices of residual connections in ResNet?
In the resnet and its variants, such as (taken from here)
Do these shortcut connections have any weight matrices (and bias) associated with them or do they simply copy the same output and transfer it ...
- 171
1
vote
1
answer
539
views
Look ahead bias in the skip connection of the Transformer-decoder/GPT2 architecture
How come the residual connection on the attention module in Decoder-Transformers/GPT2 does not cause a look ahead bias?
This is my current understanding:
GPT is similar to the decoder side of the ...
- 33
1
vote
1
answer
777
views
Max Pooling vs Average Pooling for residual/skip connections
I've implemented a CNN with skip connections; some connections skip across residual blocks with no spatial downsampling but some connections skip across blocks that have convolutions with a stride of ...
- 1,128
2
votes
0
answers
143
views
Why is the block size in ResNets usually 2 or 3 layers and not more?
Is there a reason why only 2 or 3 layers are skipped by a shortcut connection? What benefits would it have to skip more than 3 layers? I do not find an obvious answer besides that increasing the block ...
- 63
3
votes
1
answer
735
views
Is a Residual Network a Feedforward Network?
I am really confused about people comparing feedforward networks to residual networks. This is done in several papers I have read into (just one for example, first line in this paper: https://arxiv....
- 33
1
vote
0
answers
234
views
Does the 'skip' in a residual network actually happen in the backward pass?
The following is a question regarding a residual block. Let $A^{[l-1]\{t\}}$ denote the activation of the $(l-1)^\mathrm{th}$ layer of a particular fully connected neural network, given the $t^\mathrm{...
- 133
2
votes
1
answer
42
views
Residual Networks - Non-image use cases
I've been doing some research into computer vision lately and constantly come across the ability of residual networks to improve performance. Intuitively I think I grasp them, however, I struggle to ...
- 21
1
vote
0
answers
226
views
How to calculate number-of-ops used in the backward pass of the neural net in training phase?
I'm trying to study a basic model like ResNet and how many operations it does and memory usage during backward-pass. For forward pass for layer like 1x1 conv or 3x3 conv, i was able to easily compute ...
- 319
-1
votes
1
answer
121
views
Where is BatchNorm performed in ResNeXT https://github.com/facebookresearch/ResNeXt neural network?
In the original paper that described ResNeXT (variation of Resnet) at https://arxiv.org/pdf/1611.05431.pdf.
On Page-5 top right column, it says:
ReLU is performed right after eachBN, expect for ...
- 319
2
votes
1
answer
812
views
how is this Bottleneck design the same as original residual block in resnet?
This paper/link talks about resnet's bottleneck design.
It's totally not clear to me how the bottleneck design on the right is equivalent to the left-diagram and how is it reducing the parameters? ...
- 319
3
votes
1
answer
2k
views
Why does residual block in resnet shown as skipping not just 1-layer (conv + relu) but also the next weight layer?
I was reading about resnet at this link. This link and others say that residual block skips 1-layer, but then all of them show a diagram where there is an additional weight layer (i think it can be ...
- 319
1
vote
0
answers
290
views
How to pad skip connections when using transposed / deconvolutional layers
if you have a standard CNN architecture with convolutional layers there are 2 reasons why the identity of the skip connection can't be added with the current output.
1) There was pooling between the ...
- 158
0
votes
1
answer
209
views
Can residual connection affect the learning of skipped layers?
My understanding of residual connection is it could help avoid degradation by making $F(x)$, the skipped layers, close to 0 so that when $F(x)$ become redundant it won't hurt the performance. What ...
- 417
0
votes
1
answer
23
views
How do Residual layers not inhibit the learning?
My understanding of residual layers is that we can take a layer and branch it into two paths. One going the typical path for a network, and the other is an identity mapping forward over the ...
- 103
3
votes
0
answers
149
views
How harmful is a wide Dense layer after a narrow?
My CNN-LSTM EEG Keras classification model includes a Dense 'shortcut' connection for residual sequence learning as shown below; to match dimensionality, the Dense layer's set to ...
1
vote
0
answers
242
views
Why is my keras resnet50 model overfitting? [duplicate]
I have applied Keras ResNet-50 on a small x-ray image dataset. I tried making layers both trainable and non-trainable, but my model validation accuracy doesn't improve above 50%.
I don't understand ...
- 121
1
vote
1
answer
441
views
Why resolution is not important for pre-trained models
As far as I understand (and even successfully applied in Kaggle competition), it's possible to feed images of any resolution into the pre-trained model (e.g. ResNet34).
But I do not understand, why it ...
- 113
0
votes
1
answer
479
views
Residual Network: Is small deviation of layer response good? What's the point of resnet?
I have 2 questions...
In the paper Deep Residual Learning for Image Recognition, it says
We show
by experiments (Fig. 7) that the learned residual functions in
general have small responses, ...
1
vote
1
answer
646
views
ResNet 34 training with custom dataset
I am a beginner in Neural Networks and wanted to implement ResNet34 for a pet project at my workplace. Due to confidentiality issues, I do not want to use ImageNet trained weights.
I have a dataset ...
- 111
1
vote
1
answer
5k
views
How does backpropagation in residual network works?
i am studying the deep residual network currently, and i cannot fully understand how backpropagation in residual network works. Here some parts of the paper that i read.
so, how does the ...
2
votes
1
answer
1k
views
ResNet: What is the content of the second skip-connection?
I have a question regarding the second skip connection in ResNet.
Here is a part of the image of the architecture as it was presented in the paper:
As I understand the output of the pool layer gets ...
- 158
5
votes
1
answer
643
views
Can a residual neural network be interpreted as a form of ensemble learning?
I stumbled over this paper: "Veit, A., Wilber, M. & Belongie, S. Residual Networks Behave Like Ensembles of Relatively Shallow Networks. (2016).". In it, they argue that residual neural ...
4
votes
1
answer
1k
views
How to initialize weights in the presence of skip connections?
Weight initialization is an important parameter for success in large networks, in the absence of techniques such as batch normalization that reduces their impact.
There are known initialization ...
- 147
3
votes
1
answer
2k
views
How do the residual blocks prevent exploding gradients?
I am reading Roger Grosse's lecture notes on ResNet and I have a question regarding the explanation on how residue blocks prevent gradient explosion, see the screenshot below:
My confusion is: this ...
- 151
0
votes
1
answer
534
views
Residual connection in Xception [closed]
I am trying to understand residual connection in Xception.
If I am getting right, there's nothing really happen in residual connection (right figure) because it is just addition.
But, I could not ...
- 3
3
votes
1
answer
6k
views
How resNet increasing the dimension?
In the above image, It is the part of the resNet Architecture, here they have used dotted line to increase the dimension, but my question is How they are increasing the dimension?? or this dotted line ...
- 191
35
votes
5
answers
44k
views
What are "residual connections" in RNNs?
In Google's paper Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, it is stated
Our LSTM RNNs have $8$ layers, with residual connections between ...
user82135
2
votes
1
answer
2k
views
Residual network: why is each block learning residual error with respect to identity mapping?
In original version of neural network, it learns H(x) from input x, and in the residual network, it is said that learning is improved by learning only residual error with respect to identity mapping, ...
- 311
12
votes
3
answers
11k
views
Shortcut connections in ResNet with different spatial sizes
If I take Fig.3 of the paper "Deep residual learning for image recognition", and look at the following piece of the residual network:
$3\times3$ conv, 64 filters
...
- 221
1
vote
0
answers
154
views
Validation error smaller than training error in Deep Residual Learning for Image Recognition paper
At one of the examples of Deep Residual Learning for Image Recognition paper, figure 4, leftmost graph it's said that thin curves denote training error and bold curves validation error.
Throughout ...
- 111
0
votes
1
answer
422
views
How neural networks manages to learn 0 class for sigmoidal Cross-Entropy loss function using ReLU as activation unit?
Let's say we have binary classification problem: 0 vs 1. Some set of images needs to be mapped to those labels by convolutional neural network.
Main trend for now is to use ReLu as activation unit, ...
- 211
4
votes
1
answer
800
views
Are shortcut connections with stride > 1 still "identity mappings" in ResNets?
In Deep Residual Learning for Image Recognition, I am trying to understand better the "dotted shortcuts" from Figure 3, where the first convolutional layer in those shortcuts is applied with stride of ...
- 141
13
votes
1
answer
10k
views
What exactly is a Residual Learning block in the context of Deep Residual Networks in Deep Learning?
I was reading the paper Deep Residual Learning for Image Recognition and I had difficulties understanding with 100% certainty what a residual block entails computationally. Reading their paper they ...
- 6,566
11
votes
3
answers
9k
views
Why do residual networks work?
I have a few questions about the paper Deep Residual Learning for Image Recognition by
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun.
The building blocks of residual networks can be viewed as ...
- 211
3
votes
2
answers
437
views
Do ResNets from Microsoft experts represent convolutional neural nets?
I am talking about the article of Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun (Microsoft Research team):
"Deep Residual Learning for Image Recognitione (2015)"
ResNets won the 1-st place at ...
5
votes
2
answers
3k
views
How does the Identity connection in ResNets work
I am currently going through the Research paper "Deep Learning for Image recognition" by Kaiming He. I don't quite understand the concept of shortcut connections.
Suppose the input to the residual ...
- 418
15
votes
2
answers
3k
views
Are Residual Networks related to Gradient Boosting?
Recently, we saw the emergence of the Residual Neural Net, wherein,
each layer consists of a computational module $c_i$ and a shortcut connection that preserves the input to the layer such as the ...
- 990
6
votes
1
answer
2k
views
Residual network dimension changing blocks identity function
In trying to implement ResNet with bottleneck blocks for myself, I got very confused about the identity function residual blocks with different dimensions. They compared identity, conv projections on ...
- 61