I am currently facing an issue with one of the models I am working on (it's a Transducer model but that does not really matter for the question in general). The problem is that the model sometimes does not converge and I haven't found any explanation so far for this.
At the beginning I made the mistake and not use bucketing during training. This asserts that samples presented to the model are roughly the same size such that padding is minimized. After adding bucketing the model started to converge but this changes now again after I (seemingly) lift the limit for how long samples can actually be.
The result is a variable batch-size. Below is an example how the batch size gets affected based on the approximate sample length:
So, essentially, I would like to know if a variable batch-size can destabilize or mess with the optimizer. In this case I am using Adam.