This is much more of a computer-related question but I believe this requires statistical knowledge (which I'm not well versed and currently reading on the basics) so i hope you can bear with me.
I have below as factors:
G = number of groups
n(1), n(2), .., n(G) = number of records in G (n varies in each group)
There is also a constant T which is number of threads where the processes runs concurrently. In this case, T = 4.
For example, G = 5 and below are the n values for each group:
1 - 10000
2 - 8581
3 - 1500
4 - 15678
5 - 7546
Since T = 4, only the first 4 will be accommodated and processed concurrently. Group 5 will be waiting until one of the process finishes.
Assuming they finished with below elapsed times...
Thread --> n records --> elapsed time
T1 --> 10000 records --> 25 s
T2 --> 8581 records --> 23 s
T3 --> 1500 records --> 3 s
T4 --> 15678 records --> 28 s
Since T3 took only 3 seconds to process, it will be the first to finish. Group 5 will then be accommodated in T3.
T3 --> 7546 records --> 20 s
Hence, the total runtime for all 5 groups will be 28 seconds because it is the last process to finish. Group 5 will finish at the same time Group 2 finishes (23 s). Elapsed Time = 28 s
Given the above example, I am having difficulty on how to derive an estimate given the number of groups and the number of records per group because G and n can vary (e.g. input can be any number of groups with varying records per group). If this is to be solved using a statistical approach, what method should I use that best fits this case? I have read on the fundamentals about linear regression but there are other regression types as well. I also have historical data on previous runs as well.
Any ideas / insight is very much appreciated. Thank you in advance.