I have made a computer simulation, which is a model of a machine process step in a real factory. This factory keeps track of certain KPIs (key performance indicators), like utilization (% of time in use), throughput (#orders per hour), and lateness (% of orders processed too late)

My model also computes these KPIs. The values typically differ slightly, but I would like to know whether these discrepancies are acceptable.

How would I be able to check this? Is there a specific method I could use?

Example values: Utilization model: 81.5%, 79.2%, 90.2%, 87.6% Utilization actual situation: 84.7%, 77.6%, 84.3%, 85.5% We are talking about a minimum of 500 orders daily, to give you an impression, all resembled in just 3 KPIs.


Unfortunately, there is no clear-cut rules for making such assessments. It is pretty subjective how much is too much. What you could do, is to translate the discrepancies into something more material: ask yourself how much would making wrong decisions based on the model would cost your factory. For example, say that your factory produces $n$ items per hour, while your model "produces" $n \pm k$ items. This means that it is wrong by $k$ items. If it says that you are going to produce $k$ items less than you produce in real life, then it's suggesting income smaller by some amount of money. If it says that you are going to produce $k$ items more, then if you made decision to follow the model's behavior, you would invest some amount of money for items that are not going to be produced and loose the money. After re-stating problem like this you can simply ask yourself how much money would you be willing to pay for wrong simulation results. In fact, this is exactly what could happen and this is the reason why you want to validate your model. Moreover, defining problem like this it makes it easier for you to justify your model in front of the management in your factory since you would be talking about something that is very real for them. On another hand, if it appears that making decisions based on your model could cost you much, then maybe it is not a wise thing to do.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.