169

Suppose that I have a conv layer which outputs an $(N, F, H, W)$ shaped tensor where: $N$ is the batch size $F$ is the number of convolutional filters $H, W$ are the spatial dimensions Suppose the input is fed into a conv layer with $F_1$ 1x1 filters, zero padding and stride 1. Then the output of this 1x1 conv layer will have shape $(N, F_1, H , W)$. So ...


102

You're on the right track. Invariance means that you can recognize an object as an object, even when its appearance varies in some way. This is generally a good thing, because it preserves the object's identity, category, (etc) across changes in the specifics of the visual input, like relative positions of the viewer/camera and the object. The image below ...


69

A 1x1 convolution simply maps an input pixel with all it's channels to an output pixel, not looking at anything around itself. It is often used to reduce the number of depth channels, since it is often very slow to multiply volumes with extremely large depths. input (256 depth) -> 1x1 convolution (64 depth) -> 4x4 convolution (256 depth) input (256 ...


46

Notation, upper and lower case https://en.wikipedia.org/wiki/Notation_in_probability_and_statistics Random variables are usually written in upper case roman letters: $X$, $Y$, etc. Particular realizations of a random variable are written in corresponding lower case letters. For example $x_1$, $x_2$, …, $x_n$ could be a sample corresponding to the random ...


39

Convolution calculations associated with distributions of random variables are all mathematical manifestations of the Law of Total Probability. In the language of my post at What is meant by a “random variable”?, A pair of random variables $(X,Y)$ consists of a box of tickets on each of which are written two numbers, one designated $X$ and the other $Y$. ...


36

Your confusion seems to arise from conflating random variables with their distributions. To "unlearn" this confusion, it might help to take a couple of steps back, empty your mind for a moment, forget about any fancy formalisms like probability spaces and sigma-algebras (if it helps, pretend you're back in elementary school and have never heard of any of ...


34

There are couple of reasons padding is important: It's easier to design networks if we preserve the height and width and don't have to worry too much about tensor dimensions when going from one layer to another because dimensions will just "work". It allows us to design deeper networks. Without padding, reduction in volume size would reduce too quickly. ...


31

The main reason I didn't understand 1x1 convolutions is because I didn't understand how $any$ convolutions really worked—the key factor is how computing a convolution of multiple channels/filters works. To understand this, I found this answer useful as well: https://datascience.stackexchange.com/questions/9175/how-do-subsequent-convolution-layers-work In ...


25

This approximate lognormality of sums of lognormals is a well-known rule of thumb; it's mentioned in numerous papers -- and in a number of posts on site. A lognormal approximation for a sum of lognormals by matching the first two moments is sometimes called a Fenton-Wilkinson approximation. You may find this document by Dufresne useful (available here, or ...


23

It seems that these kinds of layers have a minimal impact and are not used any more. Basically, their role have been outplayed by other regularization techniques (such as dropout and batch normalization), better initializations and training methods. This is what is written in the lecture notes for the Stanford Course CS321n on ConvNets: Normalization ...


20

Indeed, there seems no good explanation in a single place. The best is to read the articles from where it comes: The original AlexNet article explains a bit in Section 3.3: Krizhevsky, Sutskever, and Hinton, ImageNet Classification with Deep Convolutional Neural Networks, NIPS 2012. pdf The exact way of doing this was proposed in (but not much extra info ...


18

Let $X$ be a standard normal random variable, and let $Y = -X$ (pointwise). Then both are a.c., but $X+Y$ is $0$ everywhere.


16

I'll first try to share some intuition behind CNN and then comment the particular topics you listed. The convolution and sub-sampling layers in a CNN are not different from the hidden layers in a common MLP, i. e. their function is to extract features from their input. These features are then given to the next hidden layer to extract still more complex ...


15

Yes, in fact, the distribution is known as the Poisson binomial distribution, which is a generalization of the binomial distribution. The distribution's mean and variance are intuitive and are given by $$ \begin{align} E\left[\sum_i x_i\right] &= \sum_i E[x_i] = \sum_i p_i\\ V\left[\sum_i x_i\right] &= \sum_i V[x_i] = \sum_i p_i(1-p_i). \end{align} $...


15

Here is my suggested answer, though I don't claim to be knowledgeable. When performing gradient descent on a linear model, the error surface is quadratic, with the curvature determined by $XX_T$, where $X$ is your input. Now the ideal error surface for or gradient descent has the same curvature in all directions (otherwise the step size is too small in ...


14

After doing some digging in the literature, encouraged by Kjetil's answer, I've found a few references that do take the geometric/dynamical systems approach to the CLT seriously, besides the book by Y. Sinai. I'm posting what I've found for others who may be interested, but I hope still to hear from an expert about the value of this point of view. The most ...


13

You are looking at a final result rather than where the convolution came from. Starting from an earlier point makes the proof easier. If $X$ and $Y$ are independent random variables with densities $f$ and $g$ respectively, then \begin{align} P\{X+Y \leq z\} &= \int_{-\infty}^\infty P\{X+Y \leq z \mid Y = y\}g(y)\,\mathrm dy & \scriptstyle{\text{...


12

I will try to explain more intuitively and in-short with illustrations! A 1*1 conv (a.k.a Network-in-Network)? let's say you input is ($n_H$,$n_W$, $n_{c_{prev}}$). You can think of (1*1*$n_{c_{prev}}$) as a single neuron(a Fully Connected network-i.e why Network-in-Network) that's taking in ($n_{c_{prev}}$) numbers in the input multiplying them with (1*1*...


12

A convolutional layer applies the same (usually small) filter repeatedly at different positions in the layer below it. E.g. if the input layer has dimensions 512 x 512, you could have a conv layer that applies the same 8 x 8 filter (specified by 64 filter coefficients), at each point in (e.g.) a 128 x 128 grid overlaid on the input layer. On the other hand, ...


12

First, let me note there is nothing special in having the coefficients of the linear combination to be less or more than one. The moment generating function is defined as $$M_X(s)=\mathbb{E}[\exp\{sX\}]$$ when this expectation exists. Considering a linear combination of independent random variables, like $2X+3Y$, leads to the moment generating function \...


11

Have you taken a look at dedicated R packages for that? Like convolve, https://stat.ethz.ch/R-manual/R-devel/library/stats/html/convolve.html


11

This happens because when you take a mini-batch, it is very very less likely (given the ratio of the proportions here) that a mini batch will contain samples of your positives at all. So it will end up learning the pattern for the negative class and after a couple of epochs, everything just gets classified as negative class. There are two possible ways to ...


11

With this answer I would like to summarize contributions of other authors and provide a single place explanation of the LRN (or contrastive normalization) technique for those, who just want to get aware of what it is and how it works. Motivation: 'This sort of response normalization (LRN) implements a form of lateral inhibition inspired by the type found in ...


11

Let's suppose we could find a (measurable) function $\chi$ defined on the real numbers with the property that $$\chi(a + b) = \chi(a)\chi(b)$$ for all numbers $a$ and $b$ and for which there is a finite positive number $M$ for which $|\chi(a)| \le M$ for all $a$. Notice how $\chi$ relates addition (which is the fundamental operation appearing in a ...


11

If we have a variable $X\sim U(0,1)$ and multiply it by $a$, then $aX\sim U(0,a)$. Assume that we're dealing with independent continuous uniform on $(0,a)$ and $(0,b)$ respectively (with $a<b$) (This assumption is not restrictive since we can obtain the general case from this easily.) Then the joint density is $\frac{1}{ab} I_{(0,a)}\times I_{(0,b)}$. ...


11

Conditioning Approach Condition on the value of $X_1$. Start with the cumulative distribution function (CDF) for $S_2$. $\begin{align} F_{S_2}(x) &= P(S_2\le x) \\ &= P(X_1 + X_2 \le x) \\ &= \int_0^\infty P(X_1+X_2\le x|X_1=x_1)f_{X_1}(x_1)dx_1 \\ &= \int_0^x P(X_1+X_2\le x|X_1=x_1)\lambda \text{e}^{-\lambda x_1}dx_1 \\ &= \int_0^x P(...


10

A procedural solution for $n$ tuple convolution is given here. Q1: The two gamma distribution convolution (GDC) in closed form is only given here $\mathrm{G}\mathrm{D}\mathrm{C}\left(\mathrm{a}\kern0.1em ,\mathrm{b}\kern0.1em ,\alpha, \beta; \tau \right)=\left\{\begin{array}{cc}\hfill \frac{{\mathrm{b}}^{\mathrm{a}}{\beta}^{\alpha }}{\Gamma \left(\mathrm{...


10

Actually I don't think this is quite right, unless I'm misunderstanding you. If $X$ and $Y$ are independent random variables, then the sum/convolution relationship you're referring to is as follows: $$ p(X+Y) = p(X)*p(Y) $$ That is, the probability density function (pdf) of the sum is equal to the convolution (denoted by the $*$ operator) of the individual ...


9

I think this is a very nice problem. If I may change notation slightly ... The Problem Let $\quad W \sim N(\mu_0, \sigma_0^2), \quad X_1 \sim N(\mu_1, \sigma_1^2), \quad X_2 \sim N(\mu_2, \sigma_2^2)$ denote independent random variables, and let $c$ denote a constant. Find the pdf of $Z$, where: $$ Z = \begin{cases}W + X_1 & \text{if } W \leq c \\...


9

I think there is some confusion about what is meant by translational invariance. Convolution provides translation equivariance meaning if an object in an image is at area A and through convolution a feature is detected at the output at area B, then the same feature would be detected when the object in the image is translated to A'. The position of the output ...


Only top voted, non community-wiki answers of a minimum length are eligible