121

Suppose that I have a conv layer which outputs an $(N, F, H, W)$ shaped tensor where: $N$ is the batch size $F$ is the number of convolutional filters $H, W$ are the spatial dimensions Suppose this output is fed into a conv layer with $F_1$ 1x1 filters, zero padding and stride 1. Then the output of this 1x1 conv layer will have shape $(N, F_1, H , W)$. So ...


66

You're on the right track. Invariance means that you can recognize an object as an object, even when its appearance varies in some way. This is generally a good thing, because it preserves the object's identity, category, (etc) across changes in the specifics of the visual input, like relative positions of the viewer/camera and the object. The image below ...


48

A 1x1 convolution simply maps an input pixel with all it's channels to an output pixel, not looking at anything around itself. It is often used to reduce the number of depth channels, since it is often very slow to multiply volumes with extremely large depths. input (256 depth) -> 1x1 convolution (64 depth) -> 4x4 convolution (256 depth) input (256 ...


42

Notation, upper and lower case https://en.wikipedia.org/wiki/Notation_in_probability_and_statistics Random variables are usually written in upper case roman letters: $X$, $Y$, etc. Particular realizations of a random variable are written in corresponding lower case letters. For example $x_1$, $x_2$, …, $x_n$ could be a sample corresponding to the ...


28

Your confusion seems to arise from conflating random variables with their distributions. To "unlearn" this confusion, it might help to take a couple of steps back, empty your mind for a moment, forget about any fancy formalisms like probability spaces and sigma-algebras (if it helps, pretend you're back in elementary school and have never heard of any of ...


27

Corresponding to any batch of data $X = (x_1, x_2, \ldots, x_n)$ is its "empirical density function" $$f_X(x) = \frac{1}{n}\sum_{i=1}^{n} \delta(x-x_i).$$ Here, $\delta$ is a "generalized function." Despite that name, it isn't a function at all: it's a new mathematical object that can be used only within integrals. Its defining property is that for any ...


21

It seems that these kinds of layers have a minimal impact and are not used any more. Basically, their role have been outplayed by other regularization techniques (such as dropout and batch normalization), better initializations and training methods. This is what is written in the lecture notes for the Stanford Course CS321n on ConvNets: Normalization ...


20

Indeed, there seems no good explanation in a single place. The best is to read the articles from where it comes: The original AlexNet article explains a bit in Section 3.3: Krizhevsky, Sutskever, and Hinton, ImageNet Classification with Deep Convolutional Neural Networks, NIPS 2012. pdf The exact way of doing this was proposed in (but not much extra info ...


20

This approximate lognormality of sums of lognormals is a well-known rule of thumb; it's mentioned in numerous papers -- and in a number of posts on site. A lognormal approximation for a sum of lognormals by matching the first two moments is sometimes called a Fenton-Wilkinson approximation. You may find this document by Dufresne useful (available here, or ...


20

There are couple of reasons padding is important: It's easier to design networks if we preserve the height and width and don't have to worry too much about tensor dimensions when going from one layer to another because dimensions will just "work". It allows us to design deeper networks. Without padding, reduction in volume size would reduce too quickly. ...


19

Convolution calculations associated with distributions of random variables are all mathematical manifestations of the Law of Total Probability. In the language of my post at What is meant by a “random variable”?, A pair of random variables $(X,Y)$ consists of a box of tickets on each of which are written two numbers, one designated $X$ and the other $...


16

I'll first try to share some intuition behind CNN and then comment the particular topics you listed. The convolution and sub-sampling layers in a CNN are not different from the hidden layers in a common MLP, i. e. their function is to extract features from their input. These features are then given to the next hidden layer to extract still more complex ...


15

Here is my suggested answer, though I don't claim to be knowledgeable. When performing gradient descent on a linear model, the error surface is quadratic, with the curvature determined by $XX_T$, where $X$ is your input. Now the ideal error surface for or gradient descent has the same curvature in all directions (otherwise the step size is too small in ...


13

After doing some digging in the literature, encouraged by Kjetil's answer, I've found a few references that do take the geometric/dynamical systems approach to the CLT seriously, besides the book by Y. Sinai. I'm posting what I've found for others who may be interested, but I hope still to hear from an expert about the value of this point of view. The most ...


12

The main reason I didn't understand 1x1 convolutions is because I didn't understand how $any$ convolutions really worked—the key factor is how computing a convolution of multiple channels/filters works. To understand this, I found this answer useful as well: https://datascience.stackexchange.com/questions/9175/how-do-subsequent-convolution-layers-work In ...


12

First, let me note there is nothing special in having the coefficients of the linear combination to be less or more than one. The moment generating function is defined as $$M_X(s)=\mathbb{E}[\exp\{sX\}]$$ when this expectation exists. Considering a linear combination of independent random variables, like $2X+3Y$, leads to the moment generating function \...


11

With this answer I would like to summarize contributions of other authors and provide a single place explanation of the LRN (or contrastive normalization) technique for those, who just want to get aware of what it is and how it works. Motivation: 'This sort of response normalization (LRN) implements a form of lateral inhibition inspired by the type found in ...


11

Let's suppose we could find a (measurable) function $\chi$ defined on the real numbers with the property that $$\chi(a + b) = \chi(a)\chi(b)$$ for all numbers $a$ and $b$ and for which there is a finite positive number $M$ for which $|\chi(a)| \le M$ for all $a$. Notice how $\chi$ relates addition (which is the fundamental operation appearing in a ...


10

This happens because when you take a mini-batch, it is very very less likely (given the ratio of the proportions here) that a mini batch will contain samples of your positives at all. So it will end up learning the pattern for the negative class and after a couple of epochs, everything just gets classified as negative class. There are two possible ways to ...


10

A procedural solution for $n$ tuple convolution is given here. Q1: The two gamma distribution convolution (GDC) in closed form is only given here $\mathrm{G}\mathrm{D}\mathrm{C}\left(\mathrm{a}\kern0.1em ,\mathrm{b}\kern0.1em ,\alpha, \beta; \tau \right)=\left\{\begin{array}{cc}\hfill \frac{{\mathrm{b}}^{\mathrm{a}}{\beta}^{\alpha }}{\Gamma \left(\mathrm{...


10

A convolutional layer applies the same (usually small) filter repeatedly at different positions in the layer below it. E.g. if the input layer has dimensions 512 x 512, you could have a conv layer that applies the same 8 x 8 filter (specified by 64 filter coefficients), at each point in (e.g.) a 128 x 128 grid overlaid on the input layer. On the other hand, ...


9

Yes, in fact, the distribution is known as the Poisson binomial distribution, which is a generalization of the binomial distribution. The distribution's mean and variance are intuitive and are given by $$ \begin{align} E\left[\sum_i x_i\right] &= \sum_i E[x_i] = \sum_i p_i\\ V\left[\sum_i x_i\right] &= \sum_i V[x_i] = \sum_i p_i(1-p_i). \end{align} $...


9

I think this is a very nice problem. If I may change notation slightly ... The Problem Let $\quad W \sim N(\mu_0, \sigma_0^2), \quad X_1 \sim N(\mu_1, \sigma_1^2), \quad X_2 \sim N(\mu_2, \sigma_2^2)$ denote independent random variables, and let $c$ denote a constant. Find the pdf of $Z$, where: $$ Z = \begin{cases}W + X_1 & \text{if } W \leq c \\...


9

Have you taken a look at dedicated R packages for that? Like convolve, https://stat.ethz.ch/R-manual/R-devel/library/stats/html/convolve.html


9

You are looking at a final result rather than where the convolution came from. Starting from an earlier point makes the proof easier. If $X$ and $Y$ are independent random variables with densities $f$ and $g$ respectively, then \begin{align} P\{X+Y \leq z\} &= \int_{-\infty}^\infty P\{X+Y \leq z \mid Y = y\}g(y)\,\mathrm dy & \scriptstyle{\text{...


8

In short, local connectivity and parameter sharing (optional). In terms of image data, local connectivity says only neurons within a local region should be connected together, which basically assumes that pixels nearby are correlated, and pixels far apart are independent. Parameter sharing means that the same set of parameters applies to different ...


8

If we have a variable $X\sim U(0,1)$ and multiply it by $a$, then $aX\sim U(0,a)$. Assume that we're dealing with independent continuous uniform on $(0,a)$ and $(0,b)$ respectively (with $a<b$) (This assumption is not restrictive since we can obtain the general case from this easily.) Then the joint density is $\frac{1}{ab} I_{(0,a)}\times I_{(0,b)}$. ...


8

There are no differences in what neural networks can do when they use convolution or correlation. This is because the filters are learned and if a CNN can learn to do a particular task using convolution operation, it can also learn to do the same task using correlation operation (It would learn the rotated version of each filter). To find more details ...


8

Forget the Gaussian part for a moment. Compare these two simple situations: A) take a coin whose two sides are marked with 0 and 1 and a die with 20 sides numbered 1 to 20. Toss the coin and roll the die --- and add the results to get a total. Consider these questions (and hints): What's the chance you get a total of 0? (you can't!) What's the chance you ...


8

Actually I don't think this is quite right, unless I'm misunderstanding you. If $X$ and $Y$ are independent random variables, then the sum/convolution relationship you're referring to is as follows: $$ p(X+Y) = p(X)*p(Y) $$ That is, the probability density function (pdf) of the sum is equal to the convolution (denoted by the $*$ operator) of the individual ...


Only top voted, non community-wiki answers of a minimum length are eligible