you say "Dropout is independent of data size" and "no need to change [hyperparams]". Do you have any evidence to support this? This seems at odds with your original answer that says I should reduce regularization and add layers, and that dropout is less important. "I would run multiple models..." I can't measure overfitting when training on all available data. That is the main point I am trying to get at, please let me know if I can clarify the original question at all.

To elaborate on the motivation, I'm primarily interested in the second scenario, using multiple past-to-future splits. This notebook shows a good visualization of how this split would work, and why the scale-up could easily be 2x or more.

Thanks Brian, thats helpful. But ideally I'm looking for specifics. So for instance if my training data doubles, should I half my dropout rate? What are the optimal number of layers to add to get the most out of the extra signal? The key thing here is that once I am using my full training data set, I no longer have a validation set to tune these parameters.

Interesting problem. To clarify, do you mean that you have one feature with cardinality of n? So if each pixel is "good", "bad" or "ugly", then you'd encode them as either .33, .67, or 1 accordingly in a single channel. Do I understand correctly? You could also try one-hot encoding them, and then each level could have it's own channel of 0s and 1s. But idk if this will work well - it probably won't if cardinality is high.

thanks @chrishmorris!

Thanks Ben, I'll try shifting the probabilities. I can't share my code unfortunately, but I'll try to find time to replicate the issue on public data, and will post an update here once I do.

One idea I had was that a different set of class weights could have a different set of optimal hyperparams. But I did a quick hyperopt with the class weights included, and the architecture didn't change much, and performance was still bad.

By the way, can you elaborate on why ROC isn't good for unbalanced data? I've heard this a few times, but have never fully understood the argument. I understand that you can have high AUC and a "low" precision (as in my case), but the axes are normalized, so it's not "gameable" like accuracy is. Better AUC should still generally mean better model, no?

Let's say my goal is to maximize precision while recalling X% of my positives (basically PR AUC). Measured by this metric, or measured by log loss, I'm seeing the same pattern - class weights hurt the model dramatically.