How do you go from paper to code?

Question

My background is in Computational Biology, although I am a Biologist by training. I just recently started my PhD, and while I have a strong interest/curiosity in Computational Biostatistics/Epidemiology and method development, I am fairly sure I won't have to develop and implement new methods. But as I said, I am curious about it.

More precisely, I still do not understand how someone goes from a research paper to e.g., an R package. I find it very difficult, not only because maybe I do not have the proper theoretical understanding of it, but also because I find programming statistical methods particularly difficult. I know how to code, I code quite a bit, but I never had to implement a method starting from the paper presenting it.

Just to give an example, I recently came upon this paper and apparently somebody implemented it in R here (file named focus_1.0.0.tar.gz). I genuinely would not know even where to start. What kind of mental model do you need to have in order to be able to translate such a framework into code?

Answer 1

More precisely, I still do not understand how someone goes from a research paper to e.g., an R package.

Like a lot of things in life, sometimes the hardest part is just getting started. A lot of people set themselves up for failure by just, well, thinking they aren't prepared enough and are going to fail, so they effectively give up before they even try.

With that said, you do have to know at least a little to get started. In the context of writing R packages based on a paper, you don't necessarily need to be comfortable with the math or theory behind the topic to get started. You might find the process of writing the package actually helps with developing that deeper understanding you eventually do want to have, and that over time you'll make changes to the package that reflect that. I actually have a very robust, user-friendly R package where I started in this situation. I have a lot of programming experience, and someone I know had this non-trivial theory he'd been developing but didn't have the ability to properly implement in a package, so he asked me to help. The details of the math and theory were mostly unfamiliar to me since my expertise is somewhere else. I didn't even know the details of creating an R package at that point. But now, after having gone through the process of creating the package, I'm at a point where I'm even able to make my own contributions and advancements to the theory and its applications.

To start, what you need to do is think about what users might want in a package. What does the paper present that potential users will find useful? This does require at least a high-level understanding of what the paper is accomplishing; presumably, you wouldn't be trying to write a package for a paper if you had no idea what the paper is about. So make a list of the things that users will want. Now assign these items to function names without worrying about the details of how these functions work internally. For example, you might have one function that does a core statistical analysis, one that calculates confidence intervals, one that calculates variances, and maybe one that does a basic visualization.

Next, what are the most fundamental inputs you need for these functions? There might be a lot of little things that are useful for convenience, tuning, or flexibility in the function, but these aren't important yet. What's important is just getting the basic functionality running; you can go back and deal with the finer details later so that they don't overwhelm you now.

Now, what are these functions returning for values? In other words, what would a user expect or find most useful in terms of results? Part of this may end up shaping or being shaped by the previous point about inputs. If you have one function that builds on the result of another, then the input of one is going to be the output of another. So depending on the situation, you might have one function that returns simple things like a numeric or a vector, and you might have another function that returns complex things like a list or class object.

Once you have a list of the functions you want, what they need for inputs, and what they produce, you now have what you need for a basic skeleton package. So start coding by creating a bunch of empty functions. Or maybe just have them print something. If you've never written a package before, this a good time to try building and installing it (locally on your computer) to make sure it works. As you work, periodically build, install, load, and test your package to make sure everything is doing what you think it is.

Finally, with a skeleton package setup, you can now start digging into the mathematical/theoretical details. Your functions act as guide to help you break down this task so that you don't try to overwhelm yourself by focusing on everything at once. Just focus on one thing and work on getting the basics of the function running. You absolutely should not be worrying about performance at this point. Same goes for other things like weird corner cases with input data. You should be focused on getting things implemented in the simplest manner possible. Once everything is running and you have a better feel for everything, then you can revisit.

Eventually, you will pick up a deeper understanding of what it is you're trying to accomplish with your package. You might find yourself realizing that there are functions you need that you didn't think of before, or maybe a couple of your functions need to be combined. You may even look back and realize there is a completely different way of doing things that is better. For my package, v0.1.0 (essentially a beta) and v1.0.0 (first production release) are night and day different. I completely rewrote the package from scratch for v1.0.0. v0.1.0 was great for learning, not only the math and the theory, but how my framework decisions felt both as a user and as a package maintainer. Turns out, it was clunky and unpleasant for both, and a complete overhaul was worth investing in. So don't worry about getting it perfect right away; take your time in the pre-release stage to try things, and preferably give some potential users a chance to try it and give feedback as well. I can't tell you how often other people's real-world data will expose weird corner-cases you never considered...

Some additional tips:

• Technical debt: sometimes there are situations where there are multiple ways to do a task. Before publishing your package, you need to think ahead what these choices mean, because if you just take the easy way, you may find it costs you a ton of time later to work around. It's best to take care of this before publishing because you want to avoid breaking things for users once they start using the package. There are tons of articles on "technical debt". Spend some time reading a few of them.
• Maintenance burden: If you intend to create an R package, please be committed to maintaining it. A huge problem I've seen with scientific software is that many people publish software/packages as a one-off task for a publication and then let it rot, which ultimately costs others a lot of time and energy when things go wrong with the software for them. Hopefully you aren't one of them. To that end, you have to be aware of how your choices in designing the package are going to affect your time commitments later. Every little thing you add is one more thing that can break, needs to be tested, requires support for users, etc., leading to more time spent maintaining your package instead of other things you want to do. So be judicious with what you add and what you cut from the feature list; if you try to add every little convenience thing that people want, you will regret it later. The same goes for thinking about what external packages you need for your package; every dependency you add is another chance for a change someone makes in their package to break your package.
• For publishing an R package, please put it on CRAN. I can't tell you how many times I've looked at the GitHub repo for non-CRAN packages (with peer-reviewed publications) only to find fundamental flaws that CRAN's "annoying" policies would have prevented. Seriously, one package I looked at changed multiple default R behaviors that many other common packages implicitly depend on. Simply by loading the package, it could completely alter the behavior of other packages in a way that was completely silent with non-obvious changes in results, and could only be fixed easily by restarting R (and never loading the package again). The commands this package was using to do this are explicitly prohibited in CRAN packages. If the author had tried to publish to CRAN, they would have caught it (or maybe they did try and decided it wasn't worth the hassle to fix just to get on CRAN...)
• Use automated testing (for R, see the 'testthat' package). This really should be a requirement for any package. Unfortunately, just because a package has unit testing doesn't mean it's being done well. It can be a bit of an art form figuring out how to do it well.
• Good documentation is necessary. And I don't just mean the typical R function references. I expect most people benefit from having a variety of tutorials/vignettes explaining concepts and giving examples. Trying to write these will actually contribute to your own understanding of the math and theory, or at the very least give you confidence in what you have learned. For R, I recommend looking at the 'pkgdown' package for this; it helps you build a website for your package that's hosted as part of your package's GitHub repo.

Some parting words: If you're going to write software for scientists, please keep in mind how important of a responsibility it is. If you really dedicate yourself and do it right, you could make a huge difference in the ability of many people to do their research more efficiently and effectively. But if you half-ass it or are lazy about maintaining it, you could screw a lot of people and cost them a lot of time, energy, and potentially money. Either way, you have the potential to make an out-sized impact on science that you might not normally have by just staying in your own little research bubble.

Answer 2

For me, I go through the maths until I understand it (and can identify methods to make the linear algebra efficient and stable). Quite often this involves writing out the maths again in the smallest possible steps (as a LaTeX document). This is often useful as a maintenance document - if you come back to the software in a year's time, you probably won't remember exactly how it works, and a step by step mathematical derivation (with annotations that explain your problems with it) is a fast way to remember it again. Usually with programming the main difficulty is not coding (e.g. working out how to tell the computer to do something) but problem solving/understanding (working out exactly what it is you want the computer to do).

The next step is to identify components of the algorithm/method that you can test individually. Don't try to implement the whole thing in one go. I quite like Heinlein's quote "when faced with a problem you do not understand, do any part of it you do understand, then look at it again".

It is also a good idea to read other people's code and work out why they have implemented it in the way that they have, and to look at how the code has been structured in the case of packages that are used by other people. Documentation is also very important, if you want people to use your code, you need to make it as easy as possible, so minimise dependencies (e.g. on third party packages) make the user interface or API simple and consistent with expectations, and provide good documentation. This takes a lot of time/effort that most of us (including me) don't seem to be able to spare ;o)

Answer 3

To supplement Dikran Marsupial's answer, the following is an articulation of a process I use personally. This is from the perspective of coding machine learning algorithms for research, rather than from a full-blown production level software engineering perspective.

Primarily, I've often found it useful to see the process of coding a machine learning algorithm from scratch as instantiating a mathematical idea:

Mathematical, pre-algorithmic stage.

Usually in machine learning, you use some kind of mathematical or formal principle to guide the development of your algorithm. It may be statistical e.g. maximum likelihood estimation, and most likely, involve a degree of optimisation. Usually, you will need to do a fair amount of mathematical derivation work, until you have what you need.

Personally, I've found that one needs both a solid under-the-hood grasp of the mathematical principles operating here, as well as absolute security in the correctness of one's derivations. Any haziness or uncertainty at this stage will compound during implementation, when one additionally has to worry about debugging. Before proceeding to the next stage, I will often use a symbolic computing package or computer algebra system, such as Mathematica, to check all my derivations, because even a single error at this stage can lead to the extremely undesirable situation of having to search for derivation and/or debugging errors.

Algorithmic.

After I've checked that my mathematical derivations are watertight, with no errors, and I'm at a level where I can specify and sketch out my algorithm on paper, I proceed to drafting formal pseudocode. Similar to the above poster, I find that forcing oneself to type this up using algorithm2e in LaTeX helps in that it commits you in a way working solely with pen and paper doesn't.

During this pseudo-code drafting stage, I start thinking about more implementational concerns, such as adapting my derivations to be as vectorised as possible, i.e. using as few loops as possible, or making use of linear algebra techniques as far as possible. However, the implementational concerns extend only as far as complying with basic advice and standard principles of algorithm design, but not as far as optimising the code.

Implementation.

Implement the pseudocode in code. The clearer your pseudocode, and the more consistent your notation, the easier this will be. Something you will probably pick for up for yourself is that it pays enormous dividends, as the above poster has stated, to ensure you are writing modular code. Split the code writing into little chunks, which you can individually test.

In most cases, there will almost always be very basic sanity checks you can do. For example, if your code uses gradients, then you can use finite differencing to assess this has correctly been computed. If you are implementing an EM algorithm, your log-likelihood will monotonically increase, so any decreases are most likely coding errors, provided your derivations are correct.

Optimisation.

I don't want to say much about this, except that it can get deep very quickly. At this stage, you time your code, or use a whole host of numerical linear algebra, matrix computation and convex optimisation tricks to squeeze as many performance speed-ups as possible.

---

Concerning the use of tools for debugging, I don't know much about the debugging tools in R as it's something I'm only getting to grips with. However, it looks like the RStudio debugger is similar to that of pdb in Python.

In the case of Python, the PyCharm IDE is one example where I've found that tools can vastly improve the debugging process. You can walk through each line of your code as it runs, and step in, step out, do side by side evaluations, as well as keep track of all variable states at once. I personally find that the visual means of accessing all variable states at once is far superior to using doing manual command line queries using pdb. Whilst I wish I knew this earlier, I emphasise this is for consideration only, and I have no affiliation with JetBrains whatsoever.

Some final things. Like most things in life, you get better through practice. There are lots of papers accompanied with code at paperswithcode.com; and there is also an ML paper reproducibility challenge also.

Lastly, if you ever find that debugging is starting to grind on your soul somewhat, I've found it helps to think of the days when people used to use punch cards to implement algorithms...