I have a dataset from a black box function, about 35K lines in a text file, with each line containing a single string from the black box function. I am building a VAE to (hopefully) model that data, with the ultimate goal of function approximation of the black box algorithm that is generating these strings.

Each string is 88 characters in length, with 64 symbols total in the corpus alphabet (A-Z, a-z, 0-9, and special characters _ and -). Here is a small sampling:





The dataset has a Shannon entropy score of ~5.317346465637096, which according to the framework I am using falls well beneath the 7.5+ score associated with compressed or encrypted data. The black box function is likely some type of integer valued routine with a weak (non-cryptographic) hashing method. I have converted the entire dataset into one hot representation using a binary label for each character in the alphabet (e.g. A = 000 ... 001, B = 000 ... 010, C = 000 ... 100 etc). This results in a dataset where each line contains 5,632 binary inputs for the VAE.

So my problems are twofold:

  1. I can't figure out a way to visualize this dataset in a meaningful way, other than perhaps PCA.

  2. I also can't figure out a way of validating the VAE output, to insure that the generative decoder is creating output within the domain of the original black box function.

In theory I guess PCA could be used to visualize both datasets, where I could then compare the two?

Any thoughts on the foregoing would be greatly appreciated, including alternatives to VAE for the black box function approximation piece (GAN? generative RBM?)



1 Answer 1


An approach for point 2 would be: keep some of the lines as test data. I call one string x. Under the learned VAE model, you have the ELBO as the approximation to the true likelihood p(x). You can use the ELBO as a surrogate for p(x), compute the likelihood for test strings and for strings, you have generated randomly. If it has learned sth, the test strings must have a higher p(x) (ELBO(x)) than the randomly generated strings.

You can also start modifying a test string digit by digit and show that the likelihood decreases for the modified digit...

But this of course a very coarse verification...

For point 1: you could try to visualize the strings as images... make gray values from the characters (for instance using their ASCII values), say 8x11 images and try to recognize patterns by eye. Then you can generate new strings from your trained VAE model, reshape to 8x11 and inspect them visually and try to see some sort of resemblance. But this has the assumption that there is some pattern that is visually recognizable, of course...

  • $\begingroup$ So in theory strings from the test set would start showing up in the generative decoder? That's an interesting idea inre: string visualization via images. I had considered maybe phase plane analysis, but haven't yet figured out a way of splitting up those 88 characters into something meaningful to plot xyz coordinates. $\endgroup$
    – tnr
    Commented Jan 10, 2018 at 16:34
  • $\begingroup$ "So in theory strings from the test set would start showing up in the generative decoder?" NO! That's not what I mean, you have to generate an enormous number of samples before you generate one in the test set. I am trying to say the generated strings have to look like the training strings. $\endgroup$
    – user166243
    Commented Jan 10, 2018 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.