I'm not sure how helpful my answer will be but I'll try to contribute some guidance. It seems like you've partially answered your question already: short layouts negatively impact performance, and long layouts are wasteful because they under-utilize animal housing. You can start with those basic guidelines and perhaps factor in the number of animals (i.e., crowdedness) of any particular batch of animals; the animals' sizes, sexes, and perhaps the type of food they're being given. You may have to use limited data for which results are known and make projections or extrapolate results for other possible ranges of conditions by using a formula of some kind.
The fact that you can get away with an occasional short layout time suggests there is a biological mechanism at play (such as: the scent of previous animals in the housing area slows down the new animals' growth temporily, but only if it layouts change frequently?) --- this would suggest another factor in your equation: number of layout periods the animals have been exposed to, and length of prior layout periods, for the batch of animals in question. You may have to dissect the conditions to come up with candidate variables that have predictive power.
I'm thinking you could model all this with a regression equation(s) where the predicted output variable is feed conversion (which you probably want to maximize) and is dependent on the above parameters, with each parameter having a weight. Or where mortality (which you want to minimize) is similarly predicted by weighted variables. Or an all-encompassing equation that figures both performance and mortality into the overall "quality" of the layout schedule perhaps.
You could also do a logistic regression model where mortality (occurring or not occurring) is the dichotomous variable of interest, predicted by the above variables. Alternatively a linear regression model with mortality rate as your outcome of interest could work.
It's also possible you could borrow from various bio-social sciences and come up with an "index" or score of the conditions (herd size, prior layout periods, etc), and use this index as a single predictor variable for a regression model of performance or for mortality rate.
Since you would be extrapolating with these regression models and theoretically predicting the results of as-yet-unencountered conditions, you should try to become aware of any biological mechanisms involved. For example in the situation I mentioned above about the scent of previous animals in the housing area, perhaps the scent of only pregnant females, or of fertile males, might influence the next batch of animals more strongly than just the scent of any animal. Or they may be biological thresholds at work such as an ideal herd size that some types of animals naturally prefer and are most productive with. (these are just examples).
I'm also thinking that using a spreadsheet might be the best way forward so that you can tweak the regressions, enter parameters of interest and set limits on these input values, and graph your results. It might be more intuitive to work with a spreadsheet at first than coding it directly in R.
As a theoretical example, your final regression equation might be: feed conversion (your predicted variable of interest) = herd size squared, minus the female-to-male ratio, plus 3 X number of "layouts" in the last six months, minus 2 X the duration of the last "layout" period (in days). Naturally it will take some time to build and tweak a regression equation and validate it with data. And changing one variable (e.g. herd size) may change the strength of the effect of other variables (e.g. sex ratio) on performance outcome. That's where creative graphing may come in handy: you could identify more than one "sweet spot" of ideal conditions that maximize performance. My advice is to measure what you can or use what past measurements that are available if possible, for example data on each batch of animals that have cycled through over the last year. That would be especially useful for logistic regression. With various statistical software can basically throw in the variables and have the computer estimate a regression equation for you as a starting point. You may also have to tell your bosses that you'd have to experimentally tweak the animals' conditions to determine, for example, the minimum "layout" time that has any effect, or the maximum time beyond which no greater negative effect is shown. That way you will avoid wild performance predictions from unusually high or low input values.
In biomedical research we use predictive modeling like this. Usually we take an established dataset, build a model from half of it and use the model as a starting point to predict outcomes on the other half. With more verification the model can be used to predict outcomes in other populations too.
I hope I've gone down the right path of reasoning for your problem and I hope this helps!