Does reinforcement learning apply to a non-deterministic black-box environment that has no numerical state space design?

Question

Scenario:

I have a scenario as follows: There is a song box. It has 2 black box modules inside it. The song box takes input 1 'edm' song. The first black box takes the 'edm' song (it could take a 'pop' song too but we feed only 'edm' songs) and produces (non-deterministically) N more song names that is a mix of 'edm' and 'pop'. The second black box non-deterministically filters out the songs. It outputs song names that are 'edm' only.

Song box:

Problem statement: Given 10 'edm' songs as initial input and this song box, in 2 hours time, find as many 'edm' songs as possible (try to maximize on the number of 'edm' songs generated). So, it is possible that 'edm' song A could generate 100 more 'edm' songs whereas 'edm' song B could generate only 2 'edm' songs in 2 hours. I need to optimally decide which songs to select to expand to get maximum rewards (reward = number of unique unseen 'edm' songs generated). I also consider feeding generated song names into the song box to further exploit them. The generator is a sparse generator (ratio of # of 'edm' songs generated/# of 'pop' songs generated < 1; sometimes 0).

What my agent does:

How I represent song names numerically:

I can't feed song names (strings) to RL model. I need numerical representation. Hence, I use a completely unrelated module (unrelated to song box). This module is a deterministic module that takes input song name and gives output the audio signal array that can be used as feature (audio signal array size is 256).

The problems I am seeing and need help with or just need a confirmation:

1. The part that I am highly doubtful of is the song box (my environment) does not provide any state to me. There is no way to formulate "what should be the numerical state of the song box that agent can map actions to". The song box just takes a song name and outputs more song names. This is why I consider the set of audio signal arrays (for 10 songs) as the state (from the module in last image). But these numbers are in no way related to my actual environment which is the song box. Does it mean my environment is unobservable and I can't apply RL here? I had asked a related question on how to craft observation/state space of such a black box environment but did not get a satisfying answer.
2. If I use RL for training, I have the constraint that I should do it in time 4 hours at max and only with 10-20 'edm' songs. I then expect it to train properly with this less initial songs in this less time. Note, that song box takes 20 seconds to process 1 song. So, in 4 hours I would be able to run only 720 timesteps if I consider picking a song, generating out of it is 1 timestep. I feel training with 10 songs (audio signal arrays) for 720 timesteps is very less for RL agent to generalize to other songs. Till now I have done some training experiments (with Actor Critic) and my trained agent does as good as an untrained agent on unseen songs. So, will it be appropriate to conclude I need lot more data and need to train for lot more timesteps to be able to generalize to any set of unseen songs? Because if much more data and training time is required, I will shift to a smart tree traversal algorithm instead of wasting time in RL (under my constraints).

I will really appreciate if my queries are addressed as this is a significant blocker in my work. Clarifications on the 2 questions would let me decide which path to pursue. I am not able to find any RL expert in my vicinity, hence asking here. In case any part is not clear, feel free to call it out in comments, I will add an update.

Answer 1

The assumption in RL is that the agent is observing some state S, and may take one of several actions A in search of a reward. If your song vector is the state, what actions are available in this state? Is there a set of actions one could take which arrive at a future state S which is the answer you seek? You also require a observer function that can issue rewards when the agent has made the correct sequence of actions. If so then RL is a suitable algorithm for your problem.

However, due to the non-deterministic environment, I don't think RL would learn very well. It's essentially mapping a path of actions to arrive at the answer from your observer function. If the sequence of actions to the solution is non-deterministic, then there is nothing to learn.

Instead I would suggest you apply an Encoder-Decoder model because it is well suited to encoding your song vector and decoding multiple strings (song names) from it.