36
$\begingroup$
  1. I would like to measure the time that it takes to repeat the running of a function. Are replicate() and using for-loops equivalent? For example:

    system.time(replicate(1000, f()));
    system.time(for(i in 1:1000){f()});
    

    Which is the prefered method.

  2. In the output of system.time(), is sys+user the actual CPU time for running the program? Is elapsed a good measure of time performance of the program?

$\endgroup$

closed as off-topic by kjetil b halvorsen, Michael Chernick, mdewey, Peter Flom Jan 27 at 13:26

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – kjetil b halvorsen, Michael Chernick, mdewey, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 3
    $\begingroup$ Just for the record, since I'm clearly far too late to change the course of this question: this is the kind of issue that I think is best suited for StackOverflow. $\endgroup$ – Matt Parker Oct 2 '10 at 17:22
  • 2
    $\begingroup$ @Matt I agree that questions about how one times a program are well suited for SO. I also agree that a literal interpretation of this question (as taken by several of the answers) would place it off-topic here on CV. There does seem to be some statistical interest in designing a timing experiment and in analyzing the results of a such an experiment though. $\endgroup$ – whuber Jan 2 '15 at 15:31
19
$\begingroup$

For effective timing of programs, especially when you are interested in comparing alternative solutions, you need a control! A good way is to put the procedure you're timing into a function. Call the function within a timing loop. Write a stub procedure, essentially by stripping out all the code from your function and just returning from it (but leave all the arguments in). Put the stub into your timing loop and re-time. This measures all the overhead associated with the timing. Subtract the stub time from the procedure time to get the net: this should be an accurate measure of the actual time needed.

Because most systems nowadays can be peremptorily interrupted, it is important to do several timing runs to check for variability. Instead of doing one long run of $N$ seconds, do $m$ runs of about $N/m$ seconds each. It helps to do this in a double loop all in one go. Not only is that easier to handle, it introduces a little bit of negative correlation in each time series, which actually improves the estimates.

By using these basic principles of experimental design, you essentially control for any differences due to how you deploy the code (e.g., the difference between a for loop and replicate()). That makes your problem go away.

$\endgroup$
25
$\begingroup$

Regarding your two points:

  1. It's stylistic. I like replicate() as it is functional.
  2. I tend to focus on elapsed, i.e. the third number.

What I often do is

N <- someNumber
mean(replicate( N, system.time( f(...) )[3], trimmed=0.05) )

to get a trimmed mean of 90% of N repetitions of calling f().

(Edited, with thanks to Hadley for catching a thinko.)

$\endgroup$
  • 2
    $\begingroup$ Don't you mean mean(replicate(N, system.time(f(...))[3]), trim = 0.05) ? $\endgroup$ – hadley Oct 1 '10 at 20:29
  • 2
    $\begingroup$ If the f() call is long then it's fine. However, if the f() call is short then any timing call overhead will likely be increasing error measurement. With a single call for system.time() over many repetitions of f() one gets to divide out the error the call until it's some infinitesimal value (and it returns faster). $\endgroup$ – John Oct 2 '10 at 16:07
  • $\begingroup$ @John: Thanks but I don't quite get what you said. I am still wondering which is better, repeating f() inside or outside system.time()? $\endgroup$ – Tim Oct 4 '10 at 5:59
  • $\begingroup$ Every call to the system.time() command has some variable time it takes to call that causes some amount of measurement error. This is a small amount. But what if f() is a very brief call? Then this error can be conflated with the time taken to call f(). So, when you call f() 1e5 times inside a single system.time() call the error gets divided down into 1e5 chunks. When you call system.time() for every f() it's impact could be meaningful if time for f() is small. Of course, if all you need is relative timing it doesn't much matter. $\endgroup$ – John Oct 4 '10 at 10:13
  • $\begingroup$ Oh, and the second part is that it would be faster to just call system.call() once. $\endgroup$ – John Oct 4 '10 at 10:14
10
$\begingroup$

You can also time with timesteps returned by Sys.time; this of course measures walltime, so real time computation time. Example code:

Sys.time()->start;
replicate(N,doMeasuredComputation());
print(Sys.time()-start);
$\endgroup$
3
$\begingroup$

Regarding which timing metric to use, I can not add to the other responders.

Regarding the function to use, I like using the ?benchmark from the rbenchmark package.

$\endgroup$
1
$\begingroup$

They do different things. Time what you wish done. replicate() returns a vector of results of each execution of the function. The for loop does not. Therefore, they're not equivalent statements.

In addition, time a number of ways you want something done. Then you can find the most efficient method.

$\endgroup$
  • $\begingroup$ mod-tip: post the second part as a comment to the Dirk's answer. $\endgroup$ – user88 Oct 2 '10 at 9:28

Not the answer you're looking for? Browse other questions tagged or ask your own question.