Newest 'residual-networks' Questions

0

votes

0answers

20 views

Are Skip connections and residual connections(when used as backbone) in CNN same?

I have a certain level in understanding how the skip connection and residual connection works and I also got an intuition that they both are similar(Correct if am wrong). But here is my question. If i ...

asked Sep 22 at 13:00

shankar ram

1

0

votes

0answers

8 views

About memory savings in invertible neural networks (INNs)

I am trying to understand how much memory (in GBs) could be actually saved when switching to an INN. This paper proposed RevGAN for efficient image-to-image translation. The fundamental block here is ...

neural-networks gan residual-networks

asked Sep 11 at 20:06

nagaK

1763 bronze badges

1

vote

0answers

62 views

Does the 'skip' in a residual network actually happen in the backward pass?

The following is a question regarding a residual block. Let $A^{[l-1]\{t\}}$ denote the activation of the $(l-1)^\mathrm{th}$ layer of a particular fully connected neural network, given the $t^\mathrm{...

neural-networks backpropagation residual-networks

asked Jun 13 at 21:11

joshuaheckroodt

1133 bronze badges

2

votes

1answer

11 views

Residual Networks - Non-image use cases

I've been doing some research into computer vision lately and constantly come across the ability of residual networks to improve performance. Intuitively I think I grasp them, however, I struggle to ...

neural-networks residual-networks

asked May 25 at 13:53

Adam Green

213 bronze badges

1

vote

0answers

46 views

How to calculate number-of-ops used in the backward pass of the neural net in training phase?

I'm trying to study a basic model like ResNet and how many operations it does and memory usage during backward-pass. For forward pass for layer like 1x1 conv or 3x3 conv, i was able to easily compute ...

neural-networks modeling conv-neural-network gradient-descent residual-networks

asked Apr 29 at 19:04

Joe Black

1897 bronze badges

-1

votes

1answer

66 views

Where is BatchNorm performed in ResNeXT https://github.com/facebookresearch/ResNeXt neural network?

In the original paper that described ResNeXT (variation of Resnet) at https://arxiv.org/pdf/1611.05431.pdf. On Page-5 top right column, it says: ReLU is performed right after eachBN, expect for ...

neural-networks residuals conv-neural-network batch-normalization residual-networks

asked Apr 14 at 1:21

Joe Black

1897 bronze badges

0

votes

1answer

91 views

how is this Bottleneck design the same as original residual block in resnet?

This paper/link talks about resnet's bottleneck design. It's totally not clear to me how the bottleneck design on the right is equivalent to the left-diagram and how is it reducing the parameters? ...

neural-networks conv-neural-network residual-networks

asked Apr 7 at 0:10

Joe Black

1897 bronze badges

1

vote

1answer

211 views

Why does residual block in resnet shown as skipping not just 1-layer (conv + relu) but also the next weight layer?

I was reading about resnet at this link. This link and others say that residual block skips 1-layer, but then all of them show a diagram where there is an additional weight layer (i think it can be ...

neural-networks conv-neural-network residual-networks

asked Apr 6 at 23:09

Joe Black

1897 bronze badges

1

vote

0answers

53 views

How to pad skip connections when using transposed / deconvolutional layers

if you have a standard CNN architecture with convolutional layers there are 2 reasons why the identity of the skip connection can't be added with the current output. 1) There was pooling between the ...

neural-networks conv-neural-network residual-networks transposed-convolution

asked Nov 22 '19 at 14:27

Peter111

1484 bronze badges

0

votes

1answer

39 views

Can residual connection affect the learning of skipped layers?

My understanding of residual connection is it could help avoid degradation by making $F(x)$, the skipped layers, close to 0 so that when $F(x)$ become redundant it won't hurt the performance. What ...

neural-networks residual-networks

asked Nov 1 '19 at 10:55

eric2323223

1472 silver badges9 bronze badges

0

votes

1answer

17 views

How do Residual layers not inhibit the learning?

My understanding of residual layers is that we can take a layer and branch it into two paths. One going the typical path for a network, and the other is an identity mapping forward over the ...

residual-networks

asked Aug 3 '19 at 0:31

Matthew Kerian

1033 bronze badges

1

vote

0answers

49 views

How harmful is a wide Dense layer after a narrow?

My CNN-LSTM EEG Keras classification model includes a Dense 'shortcut' connection for residual sequence learning as shown below; to match dimensionality, the Dense layer's set to ...

neural-networks lstm information-theory autoencoders residual-networks

asked Jul 13 '19 at 18:08

OverLordGoldDragon

4282 silver badges9 bronze badges

1

vote

0answers

213 views

Why is my keras resnet50 model overfitting? [duplicate]

I have applied Keras ResNet-50 on a small x-ray image dataset. I tried making layers both trainable and non-trainable, but my model validation accuracy doesn't improve above 50%. I don't understand ...

neural-networks overfitting image-processing keras residual-networks

asked Jun 13 '19 at 18:58

Mobassir Hossen

217 bronze badges

1

vote

1answer

55 views

Why resolution is not important for pre-trained models

As far as I understand (and even successfully applied in Kaggle competition), it's possible to feed images of any resolution into the pre-trained model (e.g. ResNet34). But I do not understand, why it ...

deep-learning convolution pre-training residual-networks

asked Jan 19 '19 at 14:25

Bi0max

134 bronze badges

0

votes

1answer

180 views

Residual Network: Is small deviation of layer response good? What's the point of resnet?

I have 2 questions... In the paper Deep Residual Learning for Image Recognition, it says We show by experiments (Fig. 7) that the learned residual functions in general have small responses, ...

neural-networks residual-networks

asked Jan 15 '19 at 12:07

user3681105

32 bronze badges

1

vote

1answer

394 views

ResNet 34 training with custom dataset

I am a beginner in Neural Networks and wanted to implement ResNet34 for a pet project at my workplace. Due to confidentiality issues, I do not want to use ImageNet trained weights. I have a dataset ...

machine-learning neural-networks residuals residual-networks

asked Dec 13 '18 at 21:22

Harsh Rawat

1112 bronze badges

1

vote

1answer

2k views

How does backpropagation in residual network works?

i am studying the deep residual network currently, and i cannot fully understand how backpropagation in residual network works. Here some parts of the paper that i read. so, how does the ...

machine-learning deep-learning residual-networks

asked Oct 9 '18 at 1:59

Mario Kristanto

112 bronze badges

1

vote

1answer

625 views

ResNet: What is the content of the second skip-connection?

I have a question regarding the second skip connection in ResNet. Here is a part of the image of the architecture as it was presented in the paper: As I understand the output of the pool layer gets ...

neural-networks conv-neural-network residual-networks

asked Aug 3 '18 at 15:38

Peter111

1484 bronze badges

5

votes

1answer

296 views

Can a residual neural network be interpreted as a form of ensemble learning?

I stumbled over this paper: "Veit, A., Wilber, M. & Belongie, S. Residual Networks Behave Like Ensembles of Relatively Shallow Networks. (2016).". In it, they argue that residual neural ...

neural-networks conv-neural-network residual-networks

asked Jun 15 '18 at 13:12

Julian Carpenter

1735 bronze badges

2

votes

1answer

134 views

How to initialize weights in the presence of skip connections?

Weight initialization is an important parameter for success in large networks, in the absence of techniques such as batch normalization that reduces their impact. There are known initialization ...

neural-networks residual-networks

asked May 22 '18 at 5:07

P-Gn

1277 bronze badges

3

votes

1answer

707 views

How do the residual blocks prevent exploding gradients?

I am reading Roger Grosse's lecture notes on ResNet and I have a question regarding the explanation on how residue blocks prevent gradient explosion, see the screenshot below: My confusion is: this ...

machine-learning gradient residual-networks

asked May 14 '18 at 14:33

Y. Pei

513 bronze badges

0

votes

1answer

356 views

Residual connection in Xception [closed]

I am trying to understand residual connection in Xception. If I am getting right, there's nothing really happen in residual connection (right figure) because it is just addition. But, I could not ...

deep-learning residual-networks

asked Apr 15 '18 at 23:29

Troy

32 bronze badges

2

votes

1answer

4k views

How resNet increasing the dimension?

In the above image, It is the part of the resNet Architecture, here they have used dotted line to increase the dimension, but my question is How they are increasing the dimension?? or this dotted line ...

machine-learning deep-learning conv-neural-network residual-networks

asked Feb 23 '18 at 17:01

Sudip Das

1811 silver badge7 bronze badges

14

votes

3answers

19k views

What are “residual connections” in RNNs?

In Google's paper Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, it is stated Our LSTM RNNs have $8$ layers, with residual connections between ...

neural-networks lstm rnn residual-networks machine-translation

asked Jan 1 '18 at 3:03

nbro

2,1131 gold badge16 silver badges32 bronze badges

2

votes

1answer

1k views

Residual network: why is each block learning residual error with respect to identity mapping?

In original version of neural network, it learns H(x) from input x, and in the residual network, it is said that learning is improved by learning only residual error with respect to identity mapping, ...

neural-networks deep-learning residual-networks

asked Oct 26 '17 at 14:23

username123

2911 silver badge10 bronze badges

8

votes

3answers

5k views

Shortcut connections in ResNet with different spatial sizes

If I take Fig.3 of the paper "Deep residual learning for image recognition", and look at the following piece of the residual network: $3\times3$ conv, 64 filters ...

deep-learning tensorflow residual-networks

asked Mar 1 '17 at 22:01

floflo29

1811 silver badge5 bronze badges

1

vote

0answers

120 views

Validation error smaller than training error in Deep Residual Learning for Image Recognition paper

At one of the examples of Deep Residual Learning for Image Recognition paper, figure 4, leftmost graph it's said that thin curves denote training error and bold curves validation error. Throughout ...

regression machine-learning cross-validation deep-learning residual-networks

asked Feb 15 '17 at 13:07

maz

1112 bronze badges

0

votes

1answer

321 views

How neural networks manages to learn 0 class for sigmoidal Cross-Entropy loss function using ReLU as activation unit?

Let's say we have binary classification problem: 0 vs 1. Some set of images needs to be mapped to those labels by convolutional neural network. Main trend for now is to use ReLu as activation unit, ...

neural-networks conv-neural-network residual-networks

asked Jan 19 '17 at 15:05

Il'ya Zhenin

1411 silver badge3 bronze badges

4

votes

1answer

545 views

Are shortcut connections with stride > 1 still “identity mappings” in ResNets?

In Deep Residual Learning for Image Recognition, I am trying to understand better the "dotted shortcuts" from Figure 3, where the first convolutional layer in those shortcuts is applied with stride of ...

conv-neural-network residual-networks

asked Dec 16 '16 at 21:48

JefferyRPrice

1413 bronze badges

12

votes

1answer

9k views

What exactly is a Residual Learning block in the context of Deep Residual Networks in Deep Learning?

I was reading the paper Deep Residual Learning for Image Recognition and I had difficulties understanding with 100% certainty what a residual block entails computationally. Reading their paper they ...

machine-learning neural-networks deep-learning conv-neural-network residual-networks

asked Nov 20 '16 at 4:51

Charlie Parker

2,7673 gold badges23 silver badges50 bronze badges

4

votes

2answers

4k views

Why do residual networks work?

I have a few questions about the paper Deep Residual Learning for Image Recognition by Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. The building blocks of residual networks can be viewed as ...

machine-learning neural-networks conv-neural-network residual-networks

asked Sep 27 '16 at 12:37

Il'ya Zhenin

1411 silver badge3 bronze badges

3

votes

2answers

402 views

Do ResNets from Microsoft experts represent convolutional neural nets?

I am talking about the article of Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun (Microsoft Research team): "Deep Residual Learning for Image Recognitione (2015)" ResNets won the 1-st place at ...

machine-learning neural-networks deep-learning conv-neural-network residual-networks

asked Sep 17 '16 at 10:14

Andrey Radoncev

334 bronze badges

4

votes

2answers

2k views

How does the Identity connection in ResNets work

I am currently going through the Research paper "Deep Learning for Image recognition" by Kaiming He. I don't quite understand the concept of shortcut connections. Suppose the input to the residual ...

deep-learning conv-neural-network residual-networks

asked Jun 29 '16 at 19:10

Newstein

3482 silver badges9 bronze badges

12

votes

2answers

2k views

Are Residual Networks related to Gradient Boosting?

Recently, we saw the emergence of the Residual Neural Net, wherein, each layer consists of a computational module $c_i$ and a shortcut connection that preserves the input to the layer such as the ...

machine-learning neural-networks deep-learning gradient-descent residual-networks

asked May 24 '16 at 6:05

rhadar

8307 silver badges18 bronze badges

4

votes

1answer

937 views

Residual network dimension changing blocks identity function

In trying to implement ResNet with bottleneck blocks for myself, I got very confused about the identity function residual blocks with different dimensions. They compared identity, conv projections on ...

machine-learning deep-learning residuals conv-neural-network residual-networks

asked Apr 12 '16 at 6:16

Pioter

412 bronze badges

Questions tagged [residual-networks]

Related Tags