2
$\begingroup$

Say I have a molecule that contains 9 atoms. For each atom, I have calculated the properties as prop_1, prop_2... prop_5 using a certain methodology. To check if the methodology was stable, I ran the same calculation for each property 10 times. For each run, I have the MAE, maxAE, and RMSE values. As an example, for prop_1, I have 10 MAE, 10 RMSE and 10 maxAE values for each atom. I am trying to figure out two things,

  1. Now, say for prop_1, I want to report a number (such as mean +- SD) for all the runs on prop_1. Is there any other statistical function that can be used other than mean+-SD?

  2. Is it justified to merge all the properties and check the stability of the methodology?

I am a chemistry major so pardon my ignorance, please.

$\endgroup$
6
  • $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Commented Aug 20 at 9:44
  • $\begingroup$ This is not clear. In the first sentence, the properties are for a molecule. Later, the properties apply to each atom within each molecule... Also you don't explain what differs from simulation to simulation or how you define "stable". Without a lot more detail, you can't expect a useful answer. $\endgroup$ Commented Aug 20 at 10:29
  • $\begingroup$ Sorry for the previous question. I have edited the question now such that it is more clear now. $\endgroup$
    – Pro
    Commented Aug 20 at 10:56
  • $\begingroup$ Say for prop_1, I have 10 runs, but the maxAE values for each run are very dispersed. So I can term it as an unstable. $\endgroup$
    – Pro
    Commented Aug 20 at 10:57
  • $\begingroup$ Could you please explain what you mean by "merge all the properties"? $\endgroup$
    – whuber
    Commented Aug 20 at 15:58

1 Answer 1

6
$\begingroup$

Now, say for prop_1, I want to report a number (such as mean +- SD) for all the runs on prop_1. Is there any other statistical function that can be used other than mean+-SD?

You're looking for summary statistics. Which ones are useful depends on the task/application at hand. I.e., on what properties you actually compute and how they behave as well as on the distribution of values you try to summarize.

For 10 data points, I'd think that a diagram with the 10 points would have an excellent ink to information ratio.

Is it justified to merge all the properties ...

You may merge properties according to their chemical meaning. When doing so, make sure the scales are appropriate. Differing dimensions are a very good indicator of properties that you should not merge.

E.g., if you have $x$, $y$, $z$ coordinates as 3 of the properties, you may meaningfully merge their prediction accuracy into an accuracy for spatial/position prediction accuracy.

OTOH, I would not merge $x$ position with $v_x$ component of the speed, nor would I merge radius with angle if position is in polar coordinates.

... and check the stability of the methodology?

If you want to check stability of your calculation method, a more direct approach is to look at the variation of the predicted values themselves (as opposed to the stability of some error function).

To check if the methodology was stable, I ran the same calculation for each property 10 times.

There are usually several (many) influencing factors that can cause variation in the results, and only a subset is covered. I'd therefore recommend to report this as stability against whatever you varied between the runs, i.e., what (potential) influencing factors your computational experiment covered.

(You may want to look up ruggedness in analytical chemistry/method validation)

For each run, I have the MAE, maxAE, and RMSE values.

Also the figures of merit should be chosen according to their meaning in the task/application context. Also look up their mathematical properties. E.g. there is nothing astonishing in observing maximum absolute error with much more variation than MAE or RMSE calculated on the same predictions. Stability (variance properties) of the figures of merit is something totally separate from the stability of your property calculation.

Side thought: If maximum absolute error is relevant, possibly also the atom for which it occurs is relevant. I.e., whether a particular atom's property is particularly hard to predict.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.