High Dimensional Visualization and VAE Validation

Question

I have a dataset from a black box function, about 35K lines in a text file, with each line containing a single string from the black box function. I am building a VAE to (hopefully) model that data, with the ultimate goal of function approximation of the black box algorithm that is generating these strings.

Each string is 88 characters in length, with 64 symbols total in the corpus alphabet (A-Z, a-z, 0-9, and special characters _ and -). Here is a small sampling:

AQDGFSiGKSJPz8fVAnwFs8LoyZ_h6EM4St_DM-JKH4G1_HR30AdzlnUZeD2UH7vUQWM_U1P0O1o_gHJPIWCPAg6Y

AQDSP_9ipM5U803t-UUAxIrX76qMkse0d5R233Ln-HZHjlGeNDn2sdi-azj6_cDP_kPTCdNRBkPBrff3Z6FWVN84

AQB0KWoS8i7KZ5An1Bq69AAQEbZFMIzJlmoGq5f1sFnNZzdihNLWk0BvJt7AU8TuFKxTaPy45ccJCIR3UqX0ST31

AQAHUDZ6Q6DcvcnWQLKYkFvDc6NKATDUuSRGTbuMqmEeKm0gPQaIBv194g3K7Mf0XQR89-k0lwlTzAnirAzTpZ75

The dataset has a Shannon entropy score of ~5.317346465637096, which according to the framework I am using falls well beneath the 7.5+ score associated with compressed or encrypted data. The black box function is likely some type of integer valued routine with a weak (non-cryptographic) hashing method. I have converted the entire dataset into one hot representation using a binary label for each character in the alphabet (e.g. A = 000 ... 001, B = 000 ... 010, C = 000 ... 100 etc). This results in a dataset where each line contains 5,632 binary inputs for the VAE.

So my problems are twofold:

1. I can't figure out a way to visualize this dataset in a meaningful way, other than perhaps PCA.

2. I also can't figure out a way of validating the VAE output, to insure that the generative decoder is creating output within the domain of the original black box function.

In theory I guess PCA could be used to visualize both datasets, where I could then compare the two?

Any thoughts on the foregoing would be greatly appreciated, including alternatives to VAE for the black box function approximation piece (GAN? generative RBM?)

TIA

Answer 1

An approach for point 2 would be: keep some of the lines as test data. I call one string x. Under the learned VAE model, you have the ELBO as the approximation to the true likelihood p(x). You can use the ELBO as a surrogate for p(x), compute the likelihood for test strings and for strings, you have generated randomly. If it has learned sth, the test strings must have a higher p(x) (ELBO(x)) than the randomly generated strings.

You can also start modifying a test string digit by digit and show that the likelihood decreases for the modified digit...

But this of course a very coarse verification...

For point 1: you could try to visualize the strings as images... make gray values from the characters (for instance using their ASCII values), say 8x11 images and try to recognize patterns by eye. Then you can generate new strings from your trained VAE model, reshape to 8x11 and inspect them visually and try to see some sort of resemblance. But this has the assumption that there is some pattern that is visually recognizable, of course...