Planar Flow in Normalizing Flows

Question

While I've read "Variational Inference with Normalizing Flows" (abstract), I don't understand about an intuition of Planar Flow.

The author defined Planar Flow as below

Let $\boldsymbol{w} \in \mathbb{R}^D, \boldsymbol{u} \in \mathbb{R}^D, > b \in \mathbb{R}$ and $h(\cdot)$ be a smooth element-wise non-linearity.
Then the following formula is Planar Flow

$$\begin{array}{c} f(\mathbf{z}) = \mathbf{z} + \mathbf{u}h(\mathbf{w}^T\mathbf{z}+b) \\ \psi(\mathbf{z})=h^{\prime}\left(\mathbf{w}^{\top} \mathbf{z}+b\right) \mathbf{w} \\ |\operatorname{det} \frac{\partial f}{\partial \mathbf{z}}|=| \operatorname{det}\left(\mathbf{I}+\mathbf{u} \psi(\mathbf{z})^{\top}\right)|=| 1+\mathbf{u}^{\top} \psi(\mathbf{z}) | \quad (1)\end{array}$$

The author said that

The flow defined by the transformation (1) modified the initial density $q_0$ by applying a series of contractions and expansions in the direction perpendicular to the hyperplane $\mathbf{w}^T\mathbf{z}+b=0$.

I couldn't understand that why the transformation (1) move the vector $\mathbf{z}$ along the direction perpendicular to the hyperplane $\mathbf{w}^T\mathbf{z}+b=0$.

Would anybody elaborate this?

Answer 1

For every $z,$ notice that the displacement from $z$ to its destination $f(z),$ given by $f(z)-z,$ is a multiple of the fixed vector $u.$ Thus, if you were to diagram the effect of $f$ by drawing arrows from a selected set of original values $z_i$ to their destinations $f(z_i),$ all the arrows would be parallel. See the right hand plot in the figure below.

Next, notice that each level set of $f$ is a union of level sets of the function

$$z \to w^\top z,$$

which are parallel hyperplanes. On any such hyperplane given by $w^\top z = c,$ for some constant real number $c,$ all the arrows equal

$$f(z) - z = u\,h(w^\top z + b) = u\, h(c + b).$$

That shows they all have common length $|h(c+b)|\,||u||$ for every $z$ on that hyperplane.

Why one might call these characteristics "planar" is inscrutable.

---

Here at the left is an example of a generic $f:\mathbb{R}^2\to\mathbb{R}^2$ from Analysis with complex data, anything different?:

On the right is a "planar flow" transformation. The arrows are colored according to the value of $h.$ The common direction of displacement is $u = (2,-1)$ and the amount of displacement varies in the direction $w = (10,-1).$

Answer 2

The equation $$ \mathbf{w}^T\mathbf{z_1}+b=0 $$ defines a (hyper)plane. The vector $\mathbf{w}$ is the normal vector. For a refresher on multivariable calculus, see here.

So what happens if you have a fixed vector $\mathbf{w}$, a fixed scalar $b$, and you plug in a different point $\mathbf{z_2}$ into the above equation, and get

$$ \mathbf{w}^T\mathbf{z_z}+b= 1? $$ $1$ isn't $0$, so obviously this new point $\mathbf{z}_2$ isn't in the same plane. But what does $1$ represent?

$\mathbf{z}_2$ is in a different plane. This new plane has the same normal vector, $\mathbf{w}$, so this new plane is parallel to the old one. It's just shifted.

The function $\mathbf{w}^T\mathbf{z}+b$ operates on all of $\mathbb{R}^d$, so you can plug in any vector $\mathbf{z}$. The output represents perpendicular distance from some prototypical plane.

Then $h$, the "smooth element-wise non-linearity" will take this scalar output and map it into another, less interpretable scalar.

Then that scalar gets multiplied to a vector. This product is added to the original input $\mathbf{z} \in \mathbb{R}^d$. If the original point was on the plane, and if $h$ maps $0$ to $0$, then nothing gets added to the original vector $\mathbf{z}$.

On the other hand, if $\mathbf{z}$ was far away from the original plane, then a significant amount $\mathbf{u}h(\mathbf{w}^T\mathbf{z}+b)$ gets added to the original input vector.