# Dimensionality reduction for visualization of high-dimensional flows

My problem concerns how best to visualize the dynamical flows in a high-dimensional space. Here are all the details:

• I've trained an RNN (GRU) $$f(x, u)$$ with 50 units to perform a continuous control task. I don't think it's very relevant to my question, but the specific task is to apply force to a Newtonian point mass in a 2D space, to move it from stationary at an initial position to stationary at a goal position.
• For the purpose of this question, there is no system noise.
• Each episode of the task lasts 100 time steps (iterations of the RNN cell), and results in a 50-dimensional trajectory of network states (i.e. an array of shape (100, 50)).
• The network's inputs $$u$$ are 1) the goal position and velocity of the point mass, which are constant during an episode, and 2) feedback about the environment state (i.e. current location and velocity of the point mass) which varies over time.
• I've been using FixedPointFinder to find the fixed points (FPs) of the RNN. This works by using the RNN to evolve a set of candidate states $$x_{\mathrm{c},i}$$ until the cost function $$(f_u(x_{\mathrm{c},i})-x_{\mathrm{c},i})^2$$ falls below some tolerance, then excluding any duplicates etc. Note that $$u$$ is fixed for this operation -- each input vector to the RNN defines a different dynamical system with potentially different FPs.
• For this task, there is usually a single FP for a given input vector to the network, and I'll assume that is the case for this question.
• I've been visualizing hidden trajectories and FPs by projecting them onto the top 2 principle components. I fit the principle vectors to the entire batch of hidden trajectories, and then I re-use these vectors to transform everything else (such as FPs).
• Almost all the variance is captured by the top 4-5 PCs.

I want to visualize the dynamical flows of the RNN for a fixed input $$u$$. My current approach is this:

• Generate a regular grid of points $$x_{\mathrm{g},i}$$ in PC space (2D), and project it up into the state space (50D).
• Calculate $$f_u(x_{\mathrm{g},i})-x_{\mathrm{g},i}$$; these are the state evolution vectors.
• Project the vectors back down into PC space and plot using quiver.

For every time step of a trajectory, I store the input vector the RNN actually received. Therefore I find one FP and one vector field, at each time step of an episode -- i.e. a trajectory of 100 FPs, and 100 vector fields.

However:

• The minimum of the 2-norm of the resulting vector field in PC space (i.e. the shortest 2-vector) does not exactly line up with the FP, also projected into PC space. I think this makes sense, as projection into the top 2 PCs won't preserve the length of 50-vectors from the state space.
• The minimum of the 2-norm of the vector field in state space (shortest 50-vector) which is then projected into PC space does line up with the FP, but only at the very beginning and end of trials. I find this more confusing, because I don't think it's due to PCA, and I can't imagine why the minimum of the norm of the vector field $$f(x)-x$$ would not line up with the state that minimizes $$(f(x)-x)^2$$. One thought I had was that the discrepancy is due to the sparseness of the grid after it has been projected into the state space, but this is hard to test with my current approach because I'd run out of memory trying to scale up a regular grid to properly cover a 50D space.

Both of these effects are visible in the following animation of a single episode in PC space:

Is there some better way of visualizing, in 2D or 3D, the flows from a high-dimensional space?

One thing I suspect will work is to generate streamlines in the 50D space, which being points rather than vectors, should project into PC space similarly to the FPs at least.

I'd also be grateful to learn of other limitations to my approach.